Add support for API version 3 in migrate module

Course! Going to have to get over the "coming soon" hump though...

You'd have to de-construct the Wordpress plugin that they put together to decipher their API. Might be worth it to just wait...

Just to add a data point re: switching to an API version...

I've seen some talk that Disqus has some negative implications for SEO.

There seem to be at least two problems:
1) If you're using the embed Javascript version of Disqus, your comments are NOT indexed on your site, but are indexed on Disqus.com
2) If you use an API plugin that handles things right, then your comments are indexed on both your site and Disqus.com (I believe), putting you at risk of taking "duplicate content" damage.

I'm no SEO expert, so I'm not sure if all of the incoming links from Disqus offer any benefits to offset these problems. Regardless, problem #2 is still better than problem #1, so I'm very much looking forward to an API version...

Thanks for the great module, Rob!

I'm not certain that duplicate content is an issue. The post the comments are contained on will include other content and the page displaying comments on Disqus will be an aggregate of numerous comments. This will likely dilute and scatter the content enough to pass. For example, if you combined two books, you would now have a third, new work.

The javascript issue may or may not be a concern. Are there examples of live Drupal/Disqus sites? I'd like to look at the source.

Thanks.

Subscribing. I'm interested to see this feature as well.

The disqus function will create an instance of the Disqus 1.1 API: http://groups.google.com/group/disqus-dev/web/api-1-1

Just installed the latest dev version, entered my Disqus API key, ran update.php, enabled Drupal comments and ran the import (though I think that's done automatically during update.php).

Everything seems the same. The comments are still dependent on the external Disqus javascript. Is there something I'm missing? Perhaps I misunderstood the benefits of using the API.

Nice, thanks for testing. Update.php moves the comment threads from the path alias to the straight node/% URLs. Did you run into troubles during the update?

The update went fine but I'm not sure what I'm supposed to see after the update. Was it necessary to re-enable Drupal comments?

Nope, the update shouldn't introduce anything new visually. The update just performed backend tasks. Well technically, you should see the new Disqus design, but that's about it. I'll do a bit more cleanup on the update, and then make a release. Thanks a lot!

I see. The original feature request said "instead of the generic JavaScript integration" which threw me off since I noticed that this latest dev version still requires Javascript from disqus.com to be inserted.

A bit confused... are there plans for integrating Disqus in such a way that the comments are actually kept and indexed on my drupal pages, instead of only at disqus.com? This seems like a big deal from an SEO perspective...

Thanks for any clarification on this issue.

No word on this? I'm ready to migrate to Disqus for comments, but the SEO thing is a deal breaker for me...

subscribing as well

subscribing

subscribing

subscribing

subscribe

subscribing

Yep +1

Love the system but was taken aback by how comments are not inserted into Drupal system (requiring management elsewhere).

Subscribing, really want the SEO benefits.

I would suggest this feature talking of migration: add functionality that gets all the comments for the current node, imports as regular Drupal comments *and* marks them as deleetd on the Disqus side.

Add such a link for every Disqus comment/thread displayed - an then the SEO problems are lifted, and you can only import comments without dubious and/or outright spam links.

Anyway, will there be ever an APi-based, 'real', not JS-powered comments? Thanks.

subscribing

With the release of the v3 api, is there any more movement on this?

Updating the title, as the SEO benefits are implied by using the API and now version 3 is public : http://docs.disqus.com/developers/api/
with PHP libs already in place https://github.com/disqus/disqus-php

Awesome! Looks like we can do more with this then the old one.

I'm not sure about the rest of the posters in this thread, but we are more interested in 6.x at the moment than 7 :)

Well, considering that the API only is used in disqus_migrate.module, which is only in the 6.x version, I suppose this does only apply to 6.x. Once disqus_migrate.module is very solid, it would be good to port it to Drupal 7 though.

http://disqus.com/api/docs/

Whats the situation here? Im very interested in getting the seo benefits. Using disqus for drupal is a no go for me otherwise but I would really like to use Disqus.

subscribing...

i am using a simple module to attach comments from disqus back to nodes.
its only a test but works now for month.
i hide the comments with javascript so that users dont get double posts and all seo stuff works fine.

i also notice that the google page preview near search results mark the text in the javascript script part of disqus comments, really nice.

i pushed my lastest version to a git sandbox:
http://drupal.org/sandbox/Haehnchen/1082368

install, generate a public_key in disqus and put it the new tab "import back" under "disqus"

by now cron imports lastest 100 posts.

Is there any reason, other than resources (time), why the disqus module for drupal 6 couldn't be rewritten to use the api 3.0?

Couln't it be done in a way that would be on the path toward a drupal 7 version? Maybe it's woth a poll, but i bet many of us will be on drupal 6 for a while.

Both Drupal 6 and 7 could get the 3.0 API... Noone has uploaded a patch though :-P .

I would if I could and if the learning curve were less steep. Sorry.

Version 3.0 of the API required PHP 5.3.x, that could be a stopper for many sites

Let's hit it in 7.x :-) .... If anyone's site is on anything less than PHP 5.3, then they should really find a new hosting environment.

Subscribing.

So as I understand it, the javascript version of Disqus creates comments that can't be indexed by Google. The Drupal module is limited to using the Javascript version. Any workarounds to getting the comments indexed? Has anybody tested the module in #36? Thanks.

Subscribe

Would converting to the API require a complete rewrite or is it a specific section of code that would change?

I'm looking at the module right now and it looks fairly straightforward in terms of implementing the javascript... but for some reason, I'm imagining the API requires a lot more code, integration, etc correct? Or is it simply a matter of calling the API and printing the result to the page?

Subscribing...

Subscribe

Question - Everyone obviously wants this for SEO purposes - but is your intention to simply output the comments and hide them using CSS? I'm a bit confused as to how Disqus comments + Drupal comments would work together. My understanding is that all we really need is to be able to constantly import the comments from Disqus into Drupal.

I have some time and interest in this. I'm going to see if I can get in touch with Haehnchen about his sandbox project.

bkosborne, ideal in my situation would be if every time a Disqus comment is made it was instantly imported into Drupal comments. The comment would then not appear through Disqus, it would appear through Drupal.

Why use Disqus if I want to load the comment through Drupal? Because I've found that Disqus is very good at filtering out spam.

Why would I want the comment imported into Drupal and to load through Drupal? First, would be SEO but the second reason would be in case I decide to move away from Disqus in the future and return to Drupal's internal comment system. That should allow for a smoother transition.

Would be great if somebody like you took on this project. Unfortunately well beyond my tech abilities. The lack of SEO optimization for Disqus in Drupal is a real bummer.

Hmm... The spam argument is a bit beyond me though. Drupal has wonderful SPAM fighting on Mollom and RECAPTCHA. I think it's over complicating things a bit to try and get that working. You'd have to do some extra work to get Drupal comments to appear without the comment submission form (not too hard, but still). You'd be confusing users as they are submitting a comment using their Disqus username and then it gets deleted automatically and imported into a different comment system?

I still think the best best is to import the comments into Drupal's comment system, but not write anything to automatically delete the comment in Disqus. The comments can be output on each node just fine, but hidden with CSS in favor of showing the Disqus comments completely. That way you get the SEO benefits + comment backup into Drupal's system should you ever abandon Disqus.

Thoughts? I'm thinking of getting started on this soon - I have a great interest moving my site's comments into Disqus but want the SEO benefits as well.

Thanks for the response. Unfortunately, a combo of recaptcha, mollom, required registration and aggressive ip bans weren't that effective. We have some pretty active sites and were watching between 30 and 200 spam comments get successfully posted on a typical day. Switched to disqus and now we're getting 1 or 2 spam comments each week.

And I agree that comments should not be deleted from disqus. I have a slight preference for them to load though drupal and have disqus hidden (as I believe I could work around the issues you astutely raised) but that's just me. And whatever the course taken, I would be thrilled because the SEO issue is by far the biggest problem at this stage.

@bkosborne
as you requested on mail i updated the sandbox module to run now fully, i hope. but drupal7 only right now. do we need a drupal6 version?

check: http://www.espend.de on which i use it right now

some fixes and features:
- since disqus limit max comments per request to 100 items i added a "pager"
- on cronjob only new comments are imported since last run time
- comment count in teaser can new display without additional javascript
- display comments in block or in content area

patch disqus module:
- provide a checkbox in admin interface to disable javascript; i can provide a patch if someone commits it.

@Haehnchen,

Nice - I need this for D6 and I want to instead sync the Disqus comments into Drupal's comment system. I'm going to modify to work as a sub-module of this module for D6, and then work on porting to 7. Hope to have something for testing mid week.

Brian

Brian, Please let me know if there's anything I can do to help from a testing perspective.

I'm making a few assumptions here based on what I've read in this thread:

• The main blocker for users using Disqus on their Drupal site is that comments are loaded via JS and SEO benefits are lost
• Users are looking to switch to using Disqus completely and not maintain some sort of hybrid system with Drupal + Disqus

I know that @sockah has a special use case, but states he could probably work around it.

I'm going to work on a new sub-module that will serve to sync comments that exist in Disqus into the Drupal core comment system on cron. It won't work the other way around - as in comments that are somehow entered into Drupal core's comment system will not be sent to Disqus (there is the disqus_migrate module for this).

Doing this will serve two benefits:

1. Comments can be output on nodes server-side via Drupal's commenting system (read-only), and hidden via CSS to gain the SEO benefits
2. Should a user chose to abandon Disqus, it's becomes very easy to just un-hide the comments, re-enable writing comments, and disable the Disqus module.

I'm going to use the sandbox module listed above as a guide of sorts... will keep you all updated...

An update:

I've more or less finished the overhaul of the disqus_migrate module, with my code for "syncing" being used to both sync and import comments (replacing the code that exists for the current import functionality). I've abandoned the idea of a submodule after realizing the code I wrote could be integrated into the migrate module.

The export portion of the module has also been replaced and as mentioned here #1095194: Disqus Migration module revamp and here #1224680: WXR Export to Disqus Migrate, will simply export Drupal comments in a WXR file instead of using the API. The portion of the API for creating threads/posts is actually in beta right now anyway. Perhaps in the future we can work to have this use the API as well.

I'm performing some tests at the moment before I commit the changes to the dev branch for 6.x so others can test it as well. My biggest concern is ensuring that the following use case works as intended:
- User wants to switch over to using Disqus from Drupal comments
- User uses module to export existing Drupal comments to Disqus
- User turns on import syncing, bringing in comments submit thru Disqus back into Drupal, for SEO reasons
- No comments are duplicated in Disqus or in Drupal

The last bit there is the most difficult part, but I think I have things worked out in that regard. Look for the commit in a couple days.

Updating the title, and marked #1095194: Disqus Migration module revamp as a duplicate of this to focus the discussion in once place.

A lot more work has been done. I get very close to committing to dev, and then I think of something else that needs to be taken care of. I've added support for exporting comments via the API again (revamped), as well as exporting via XML as discussed previously.

I'm mostly just testing now with real comments from one of my sites... making sure that exporting and importing and syncing works well with the API. One thing I'm running into is how to handle syncing comments that are still "pending" in Disqus or have been deleted in Disqus.

For what it's worth, a small bit of support, thanks again for taking this on! People like you keep the Drupal community thriving!

I just commit the overhaul of disqus_migrate to the 6.x-1.x branch. You should see it on the next update. If everyone who wanted this functionality could please test this, that would be great. It's probably a good idea to create a duplicate of your database and run exports/imports on the duplicate, just to be safe for now.

I've run a bunch of testing myself and run into just a few small problems. Some things left to do:

• Add support for SSO when exporting
• Update the README/documentation to explain how to use the migrate functionality
• Add handling for comments awaiting approval/comments deleted when importing comments

That last one there is the most important... Hopefully get to that this weekend.

Brian

Hi Brian,

I've read through a couple of the threads on the development of the updated migrate module - thanks for all of your work! Just wanted to check in to see if you were able to update/post the README/docs on how to use the updated migrate functionality?

For our use case, the client has an existing D6 website using the standard Drupal comments that will transition to using Disqus while importing all of the old comments on nodes that were there previously.

I was going to test on my end with a site that has over 12,000 comments on my dev environment - if you could give me a basic workflow of what I would need to do to try it out? My understanding is I'll use the export to WXR, then import to a test Disqus forum. I'll gladly provide feedback here on how it went.

Many Thanks!
Chris

Chris,

Thanks for your offer for testing. I've been back and forth emailing with Disqus. The original re-write I did made use of a soon to be deprecated authentication method for accessing an API for a forum. The new method uses the OAuth method, so I spend some time integrating with that. I got everything working just fine, except for exporting posts themselves. I just found out yesterday from them they do not allow creating posts anonymously for a forum using the API.

From what I've gathered, the only way to create posts is to post them as the authenticated user, which would be your admin account with Disqus... but that requires that all of the posts have your name and email associated with them. No go.

The API doesn't really seem designed for exporting comments, and Disqus have instead suggested I just use the WXR method.

I'd still like to sit on this a bit more and continue talking to them & the Disqus developer group before moving forward. I want the implementation to be the best it can be.

I hope you can all remain patient as I work through this! I myself would like to export all of my comments on my personal site to Disqus but need this functionality as well :)

Brian

bkosborne,

Is there anything I can do to help? I've exchange a few emails with Disqus about this in the past but I only got dodgy answers and eventually ended up being completely ignored. I almost gave up on it but I'd love to be able to keep it on my site, so if there's something I can do to help, let me know.

I'm ready to give up on exporting via the API. It's really a shame, because exporting via API makes IMPORTING/syncing comments much easier to handle.

Now comments will have to be exported via XML, and then the date the last exported comment will have to be remembered. Then when I use the API to import/sync comments from Disqus back into Drupal, it will only sync comments since that last date.

It's not an ideal situation, especially if there are a ton of comments being posted a lot. Borfast, if you have some decent PHP experience, it would he helpful to look through the WordPress implementation of this (Disqus developers made it) to see how they do this. I assume they are using the old method of authentication and not OAuth

EDIT: Please ignore most of this as I was looking at the version 1.9 of this module instead of 1.x-dev. I definitely need to get some sleep...

The WP plugin uses an API 1.1 implementation that comes along in the package, pretty much like the one that already ships with our Drupal module. I've fired up an interactive PHP console and have been playing around with the Disqus class, calling methods and seeing what I get, and I believe we have everything we need in order to implement something like the WP plugin does.

If my eyes haven't been tricked by my sleepiness (2am here...), what the WP plugin does is:

• get the ID of the latest Disqus comment it has in the database;
• use the API implementation to get all the comments posted after that latest comment;
• store all the new comments in the database;
• store the ID of the new latest Disqus comment.

It seems that the API implementation takes care of the authentication details, so we shouldn't need to worry about those.
Is this what the Drupal module is doing? I didn't check because I need to get some sleep but I'll be back at it tomorrow.

There seems to be an official PHP implementation of the Disqus API (which also doesn't use OAuth but works like a charm) but its licensed under the Apache license, so in order to use it in this module a user would have to download it seperately, since the Apache and the GPL licenses are not compatible... :(

EDIT: Ignore me please, I was looking at old docs. The new ones clearly state the API supports OAuth. But then again, we don't really need OAuth for this module's purposes, do we? And now I'm really off to bed, before I say any more rubbish.

And before finally going to bed, the Disqus API doesn't seem to support OAuth. From http://docs.disqus.com/help/53/:

Disqus does not currently offer public authentication, though we do have plans on supporting OAuth in the near future.

Which leads to my last question for the night: what did you mean by "the old method of authentication and not OAuth"?

subscribe

bkosborne, from what I've seen, everything is in place to make use of the v.3 API, so I must be missing something.

Could you explain what exactly is the problem with the export and import functionality?

subscribing

@borfast, if you look here, it describes the "legacy" method of authentication, which is what the module is currently using. I have written changes into the module change from using the legacy method to instead using OAuth, but ran into some other issues in that I can't make certain API requests like I was able to with the legacy method.

I didn't want to push forward with the old method knowing that it could be deprecated soon. I'm going to revisit this one last time this weekend and just push the changes that make the most sense. Clearly a lot of people are waiting on this!

@bkosborne, don't the folks from Disqus have something to say about the issues you encountered? Because if they're pushing for deprecating the legacy authentication mechanism and rely on OAuth instead, they're going to need to make sure it works without any problems.

I don't quite get why they would deprecate the legacy auth mechanism: it's much simpler than OAuth, which makes it perfect for simple integrations, instead of having to leverage the OAuth beast for even the simplest thing.

Okay, I just sent them another support request with all the details again.

I tried going back to using the legacy method, and am now getting an error from them "There was internal server error while processing your request".

This is extremely frustrating. Hopefully they get back to me within a day or so. If not, I'm just going to commit the changes to 6.x-1.x-dev and put up some warnings indicating it is experimental and not fully working. That way at least others can toy around with it.

Good news, sort of? They acknowledge that I have to use the old authentication method "for now"... which means that they will probably eventually support creating anonymous posts without it. Right now I'm working on some internal server errors that I'm getting back from Disqus. Soon!

@bkosborne, in that case, why not use the official PHP implementation that I mentioned before? I tried some of its functionality in a script and it worked perfectly.

I neglected to use that because it requires PHP 5.3

From a quick code analysis with this neat little tool, the actual minimum version is 5.2, which is only needed for json_decode() but an alternative "json.php" implementation is also available (which doesn't need PHP 5.2), for when json_decode() is not available, which is included right at the top (line 25) of disqusapi.php.

Apart from that, it needs PHP 5.1 for curl_setopt_array(), which could easily be replaced with several calls to curl_setopt(), and date_default_timezone_set(), which can be replaced with a call to ini_set() and set the INI setting "date.timezone" to the necessary value.

The next lower version requirement is PHP 5.0 for stream_get_contents(), which I suppose could also be replaced with file_get_contents(), if one really wants to lower the minimum required version to 4.

This effectively makes PHP 5.1 the minimum required version without any code changes needed, which I think is already perfectly acceptable, but with some work we could lower that to PHP 4.

@borfast,

Thank you for taking the time to do what I should have done myself from the start. That's an awesome tool you found there (bookmarked!). I'll work on swapping out my "custom" integration with the official API. I actually considered branching off the migrate portion of the module to a completely different module today, but I don't want to hold up development any more. I'll write an update later tonight or on Sunday.'

Update: Sort of got this working. For one thing, I had to hack the identifiers.json file a bit to allow for getting the details of a thread by passing thread:ident and thread:link, but that was a minor change. I'm running through attempting to export 400+ comments on one of my sites but running into a few bumps. Got to right try catch blocks. Anyway, the swap was pretty painless. I'd expect to have functioning code pushed up to 6.x-1.x-dev by Sunday night

Ran into the same exact problems with the their API implementation as I did with mine, so that was not the issue. I can export somewhere between 50-100 posts, and then I get stopped with an Internal Server Error from Disqus. Here is what they had to say about it once I told him that I was getting the error when attempting to export posts:

"Thanks for waiting, we were checking on this for you. Unfortunately it doesn't look like we have support for unauthenticated, anonymous posts with the API yet. Our developers agree that this use case should be handled so we've included it in their notes, but for now it won't work. Apologies for any confusion, just let us know if we can help with anything else."

I don't know what's going on here... First they told me it was just OAuth that didn't support that, but now perhaps the API in general.

I pushed code up on 6.x-1.x-dev just now. Exporting via XML file seems to work, as does importing & syncing. Remember to run update.php. Please feel free to test this implementation, specifically trying to get export via the API working. I posted some basic instructions on the project page.

The first thing that comes to my mind is kind of a hack: have you tried exporting the comments in batches of, say, 50 at a time, so that the Disqus server doesn't bail out with that error?

I'll try to give the new code a go and let you know how it works.

Once I get the error, I can't get it "unstuck". One thing I haven't tried is to delete the local comment that it's erroring on and seeing if it was the comment that was the issue. I won't have time to look at this again until tomorrow night.

Given the amount of work needing to be done... I think the best bet for an overhaul to this module will be through some form of sponsorship from either a Drupal shop needing this additional functionality OR from Disqus directly. After all, if Drupal powers 2% of the web, they stand to benefit by not keeping us out of the conversation! :)

FYI the D7 version has moved to a sandbox
http://drupal.org/sandbox/dwkitchen/1483518