Author Topic: Macro to automatically download and format DU posts  (Read 3330 times)

0 Members and 1 Guest are viewing this topic.

Offline GOP Congress

  • Hero Member
  • *****
  • Posts: 2334
  • Reputation: +274/-113
Macro to automatically download and format DU posts
« on: November 13, 2011, 01:03:06 AM »
I am working on an Excel sheet that will download any DU comment thread automatically into an Excel sheet. Once downloaded into the sheet, the user can then click on as many messages as he/she wishes. Once this is done, a form opens up and the user can comment on the posts. Finally, the sheet will automatically format the text and save it in the clipboard, allowing the user to paste it into a comment box on Conservative Cave. For you DU trolls, this would cute your time cutting, pasting, and quoting several times and just get right to the fun part: commenting on the idiots.

I need someone to help me test it when it's ready. Preferably someone who posts on the DU thread a lot and can therefore give me the best help, for instance in showing me the best format, etc.

By the way, the functionality to download any thread and paste all comments onto a sheet, one row per comment, already exists, but I don't want to release it yet. Is this something people here would be interested in?
"The main purpose of the Democrat Party and the Left is to destroy the United States, transform Western Civilization to a tribal-based dystopia, and to ultimately kill all conservatives and non progressives." - Jonah Kyle

Offline BattleHymn

  • Hero Member
  • *****
  • Posts: 8750
  • Reputation: +974/-63
  • Not right, but not left, either.
Re: Macro to automatically download and format DU posts
« Reply #1 on: November 13, 2011, 01:04:43 AM »
I would use something like that. 

Do you know if it will work with DUmmie3?

Offline GOP Congress

  • Hero Member
  • *****
  • Posts: 2334
  • Reputation: +274/-113
Re: Macro to automatically download and format DU posts
« Reply #2 on: November 13, 2011, 02:07:10 PM »
I would use something like that. 

Do you know if it will work with DUmmie3?

I'm in the cold on this one. What is DUmmie3? Is it an existing program that downloads comments?

My program would be specific to the Cave needs, such as formatting in the comment box, etc.
"The main purpose of the Democrat Party and the Left is to destroy the United States, transform Western Civilization to a tribal-based dystopia, and to ultimately kill all conservatives and non progressives." - Jonah Kyle

Offline BattleHymn

  • Hero Member
  • *****
  • Posts: 8750
  • Reputation: +974/-63
  • Not right, but not left, either.
Re: Macro to automatically download and format DU posts
« Reply #3 on: November 13, 2011, 03:21:35 PM »
DU will be changing their format in the near future, to DU3.   

Offline GOP Congress

  • Hero Member
  • *****
  • Posts: 2334
  • Reputation: +274/-113
Re: Macro to automatically download and format DU posts
« Reply #4 on: November 13, 2011, 07:11:48 PM »
Is there a beta anywhere? Or can you post a link to any story with regard to the new version?

Sounds like they want to upgrade so they can't get copied too easily, like DUmmie FUnnies and conservative sites like Freeper and the Cave. I am a sports bet database programmer, and many sites are migrating to a "generated copy" module which prohibits screen scraping, and even copying via screenshots/OCR to avoid copy/paste issues. Not sure if this is what they are doing, but it would probably be in their best interests considering their posts are laughed at on other sites more than chaeered on their own.
"The main purpose of the Democrat Party and the Left is to destroy the United States, transform Western Civilization to a tribal-based dystopia, and to ultimately kill all conservatives and non progressives." - Jonah Kyle

Offline BattleHymn

  • Hero Member
  • *****
  • Posts: 8750
  • Reputation: +974/-63
  • Not right, but not left, either.
Re: Macro to automatically download and format DU posts
« Reply #5 on: November 13, 2011, 08:50:58 PM »
The beta access is limited a few people so far.  I can't answer any of your questions, since I don't have access to it with my moles.  Maybe someone else can chime in with more info?   

Offline GOP Congress

  • Hero Member
  • *****
  • Posts: 2334
  • Reputation: +274/-113
Re: Macro to automatically download and format DU posts
« Reply #6 on: November 14, 2011, 12:30:00 AM »
No biggie. DU lurkers have already stumbled over each other in their zest to report this thread to Pitt anyway. So I'll steal one of their gifs:

"The main purpose of the Democrat Party and the Left is to destroy the United States, transform Western Civilization to a tribal-based dystopia, and to ultimately kill all conservatives and non progressives." - Jonah Kyle

Offline GOP Congress

  • Hero Member
  • *****
  • Posts: 2334
  • Reputation: +274/-113
Re: Macro to automatically download and format DU posts
« Reply #7 on: December 20, 2011, 06:15:42 PM »
Sorry, been awhile since I broached on my pet project.

It looks as if DU3 is actually EASIER to do :) Everything parses out nicely and they don't use anti-scraping technology because I suspect they want to stay google-search friendly.

However, I'm on a couple of hard projects related to work so I won't be able to get to it until next year, hopefully around the primary. Once I finish it, though, it will be da bomb.

"The main purpose of the Democrat Party and the Left is to destroy the United States, transform Western Civilization to a tribal-based dystopia, and to ultimately kill all conservatives and non progressives." - Jonah Kyle

Offline GOP Congress

  • Hero Member
  • *****
  • Posts: 2334
  • Reputation: +274/-113
Re: Macro to automatically download and format DU posts
« Reply #8 on: August 11, 2012, 01:46:03 AM »
OK, it's been a few months but I have a prototype...and "alpha-plus" (or "beta-minus") version. It works swimmingly for me, but now I need a few people whom it would benefit.

To recap: I have a Democratic Underground Thread grabber and formatter. It works as follows:

1. If you are on DU site, you can just copy a link, either off the address bar in the actual thread, or by right-clicking the link to the thread itself.

2. Open the form that shows your copied link, OR select a DU link from the "Latest Links" listbox.

3. Excel sheet opens with the thread in an excel sheet. The original post is on top and is automatically selected (and cannot be deselected). You can peruse all the replies, then click ONLY the replies you want to bring over to the Cave.

4. A new form opens which will allow you to review all the posts you want to bring over to the cave. In addition, you can comment before and after each DU comment.

5. Once you are satisfied, click "Review Post."

6. This is where the magic really comes in. The post is automatically formatted into a single message post, with the following:

  • The title of the OP and the link to the thread is automatically included at the top of the post.
  • All DU messages are formatted in Quote boxes.
  • All Bold, Italic, Underline, and Strikethrough html are automatically converted from DU htlm to Cave BBS.
  • Likewise, all html links, picture links, and youtube links are automatically converted from DU html to Cave BBS.

7. Copy the full post and paste it into a new thread on Cave in the DUmpster section.

8. Click PREVIEW. Review the post and clean up anything or add / delete anything you like.

No more cutting, pasting, and individually formatting ad adfinitum.

Let me be very clear: There is NO AUTOMATION that involves either DU or CC in any way, other than accessing DU's thread list and subsequent thread selection (which would be done manually anyway). The only connection CC would have with the application is that the user would copy the text off the app and paste it into the CC's reply box. The app does NOT access CC in any way. I hope that's clear enough.

CAVEAT: As it stands, this program does NOT violate the DU Terms of service or Copyright concerns. So there is no worry about legal repercussions, as long as certain copyright infringements are not broken. In general, this program generates a "Fair Use" format, and there is no current DU restriction against having a program access the items.

If you want to give it a whirl, you have to have Microsoft Excel, any version from Office 2003 and up (except for MAC office 2008). You also must allow macros, as the program runs on VBA, and of course you need to be on the Internet for the program to work.

Just private message me. I'm limiting this to only a few people, and only those who've been here awhile and have posted significantly in the DUmmy forum, or to anyone the Admin authorizes.
"The main purpose of the Democrat Party and the Left is to destroy the United States, transform Western Civilization to a tribal-based dystopia, and to ultimately kill all conservatives and non progressives." - Jonah Kyle

Offline Kyle Ricky

  • Hero Member
  • *****
  • Posts: 7596
  • Reputation: +614/-1086
Re: Macro to automatically download and format DU posts
« Reply #9 on: August 16, 2012, 02:11:49 PM »
Will this work for Office for Mac 2011? If so, I would like to try it out.

Offline GOP Congress

  • Hero Member
  • *****
  • Posts: 2334
  • Reputation: +274/-113
Re: Macro to automatically download and format DU posts
« Reply #10 on: August 20, 2012, 12:20:05 AM »
Sorry for the delay on the answer.

I believe it should work for all Access versions after 2003, EXCEPT for Office 2008 (Mac). It uses VBA which had been taken out because of coding difficulties, but has returned to 2011.

I just had another glitch :argh: but I should have it done tonight. Thanks.
"The main purpose of the Democrat Party and the Left is to destroy the United States, transform Western Civilization to a tribal-based dystopia, and to ultimately kill all conservatives and non progressives." - Jonah Kyle

Offline Kyle Ricky

  • Hero Member
  • *****
  • Posts: 7596
  • Reputation: +614/-1086
Re: Macro to automatically download and format DU posts
« Reply #11 on: August 20, 2012, 12:43:11 AM »
Thanks, let me know. I will try it out.