I too am interested in getting AWS support developed. Here's a first cut of the EC2 API. I've tried to follow your lead in coding style but needed to make a few hacks to aws.module to support Amazon's Query interface as well as REST.

Unfortunately EC2 and S3 don't support REST, and I'd rather not get into SOAP unless needed. I'd also like to parse the output in the API rather than have the caller do it, but wanted to ask your opinion first. If this works for you I'll do a Query-based SQS API too.

Cheers,
Simon

CommentFileSizeAuthor
#6 aws_ec2_api.patch18.68 KBAnonymous (not verified)
#5 aws.host_.patch659 bytesAnonymous (not verified)
aws.module.patch5.21 KBAnonymous (not verified)
aws_ec2.patch18.39 KBAnonymous (not verified)

Comments

dragonwize’s picture

Status: Needs review » Needs work

This is awesome Simon.

Do not worry about changing aws.module. I fully expected it to be changing as the individual services are built out.

That being said, the switch on the host header won't work because (at least with S3) you can use a cname redirect with your own domain. I think it would probably be best to separate each API type out to its own function. ie aws_restRequest, aws_queryRequest, etc. Then each sub-module can implement the API type(s) it needs to.

I see that you are are wanting to remove rest_client dependency. I have thought about this myself and am currently undecided because I don't know enough about the other services yet. I would like to remove the dependency but for services that will be transferring files, that is not possible. From what I know about EC2, it uses S3 to for file storage. Is this correct? I also would like to keep the return object from each of the service modules as close to the same as possible so we don't have many different APIs.

That leads to your question on parsing the return. I originally planned to parse the return for the user. I removed it because I have been trying to keep the modules PHP4 compatible and (again from the S3 perspective) return bodies are in XML. As I said before I want to provide an uniform API. XML parsing between PHP4 & PHP5 is not uniform. From my knowledge PHP4 XML parsing is not a easy thing like PHP5's SimpleXML. Also I use PHP5 exclusively and don't plan on installing 4, so someone else would have to write and update the PHP4 parsing.

A couple ideas I've been thinking about:
1. I can make the rest_client return mimic the drupal_http_request as much as possible(which I will probably be doing anyway). And don't process the request.
2. We can process the requests and create common custom return API for the modules, if it can be done acceptably.

It has been a long day and I don't have time to review the rest now. Great work so far. I will review the rest tomorrow and probably start committing it.

Anonymous’s picture

Dragonwize, thanks for the positive reaction. My code comment about removing the REST dependency was just because I used one of the rest_client's helpers in the query code. I'd started implementing a query_client module to follow the same abstraction, then realised it would only have a couple lines of code. I only spent enough time on changes to aws.module to get it working. Once you get a bit further with this refactoring aws.module would probably be a good idea.

EC2 returns xml too. I agree about the xml parsing and having to support PHP4 & 5 nicely. I wonder if the solution used by http://drupal.org/project/importexportapi would work? It uses a lightweight parser that could be bundled, (http://minixml.psychogenic.com/)

Cheers

dragonwize’s picture

I don't think using that xml parser is acceptable to me. Parsing xml that way is very slow and doesn't offer the speed and power SimpleXML does. So while I want to support PHP4, I don't want to hurt PHP5 to do it.

My main objective is to leave module developers room to make decisions best for their modules. These modules do not do anything by themselves, they are helpers for module developers. So I want to try to leave as much flexibility and power to those developers without forcing them down a road.

Also D7 will be around in a year and PHP4 support will be gone. Knowing this, if we can create an API that doesn't change because of that I think it will be better.

This is why so far I've decided to leave the data handling to the module developer. Maybe we can provide some helper functions that will parse the response for you if you decide to use them but not force the developer to have their response parsed. That would allow us to have a PHP4 & PHP5 parser function. If the developer decides to be PHP5 only they can use the helper if they wish (which would only be a line or two of code) or if the developer thinks their module will benefit from PHP4 support and doesn't mind the hit on PHP5 they they can use the PHP4 parser function. This also leaves it open for them to handle the response as they see fit without helpers.

dragonwize’s picture

Simon, I have reviewed and committed with changes your query API modifications to the AWS base module. I remove some EC2 specific items as anything specific to EC2 should be in the EC2 sub-module. I also modified the functions and parameters to better match the use of the Query API.

Please look it over and let me know if you see any issues. If it looks good we can mod your ec2 module to match.

Thank again for the work. Gave you credit in the commit log :)

Anonymous’s picture

StatusFileSize
new659 bytes

Thanks, the changes look very good to me. There's just one small mistake that needs fixing.

I'll send a patch for the EC2 api later today or tomorrow once I make some progress on the EC2 Admin GUI and can test it properly.

Anonymous’s picture

Status: Needs work » Needs review
StatusFileSize
new18.68 KB

Dragonwize, I've completed the EC2 management GUI along with the EC2 API and an AWS XML API helper module that uses minixml.
I would really have preferred the API to parse the output into something usable in the API functions where it understands the context. I don't agree that "user-land" code should need to know about Amazon's data structures. Will you reconsider?

I chose minixml as it works for everyone and performance is not an issue for the responses that EC2 needs to parse. SimpleXML support can easily be added and I'll probably do it for the SQS management GUI that I'm working on.

I've attached the patch to add the EC2 API. Should I also send a patch for the XML API or create a new project for it? I'll create new projects for the EC2 and SQS management interfaces.

Cheers,
Simon

dragonwize’s picture

Title: EC2 support » aws_ec2 module
Status: Needs review » Reviewed & tested by the community

I will commit this module. You have proven good work and a willingness to help out. I will gladly assign you as a maintainer if you would like, do you have a CVS account?

You are welcome to commit your other modules to this project as long as they are completely for AWS and NOT using AWS to achieve something else. Of course do keep it as a separate module even if it is in the project.

As for the XML processing in the module, can you give me some examples of what you are wanting to do? From my experience with the services so far, there isn't anything to process any further than the XML response returned by Amazon.

Anonymous’s picture

Good news - yes, my cvs account is 'simonc'.

I'll start a separate issue for xml and we can decide what to do from there.

I think it's probably best keep the admin interfaces separate as I intend to hook them into some other Drupal modules (specifically for mass provisioning)

Cheers

dragonwize’s picture

Status: Reviewed & tested by the community » Needs work

I have made a few minor changes in the naming and documentation, other than the patch is in tact and committed.

However, I believe there are areas of improvement in this API. We need to ensure that the entire functionality of the service is available through these functions. Such as setting the API version, etc. You have many things hard coded and/or not available for module developers to modify.

Also, a few of your functions have many parameters. From the EC2 documentation and your documentation of the parameters I think that many of them would be better combined into arrays. This helps to remove having to give values for items that you don't want to use when calling the function and cleans up the functions.

I am not trying to be critical of your work, I appreciate very much. But I think that we should put a good deal of thinking into these function calls and APIs before we release. Because once we release it will be much more difficult to seriously change them when other modules are dependent. Leading to a backwards compatibility nightmare. Obviously, the functions and API will change in the future, I just want to minimize that as much as possible by putting thought, effort, and review into it before hand.

Anonymous’s picture

I agree, version and other things should not be hardcoded and should be moved out as a parameter.

Changing the parameters into an array also seems to make sense. One thing I'm not so sure about it the array key names. Should we use Amazon's names (makes our life easier and relates directly to Amazon's documentation), or change to use Drupal's variable (and function) naming conventions?

Another thing is that by using function parameters programmers get instant feedback if they've not called the function with the correct parameters. With arrays we would need to check for this. Do you think the API ought to do validation or should we just pass to AWS as is?

I'll re-do the SQS API & module to go with your suggestions and aim to do the same with EC2.

Cheers

dragonwize’s picture

Yes, I think we should use Amazon's names for the keys. Case sensitivity is what I am not completely sure on. For the REST services, they use HTTP headers, which are NOT case sensitive. So I have ensured that key case matches by converting all keys to lowercase. That way it doesn't matter what case the extending module uses. Is the Amazon Query API parameters case sensitive?

I would try to group the parameters into arrays where they make sense. Grouping things that belong together. If there are required parameters and they make sense outside of the array, leave them outside. But yes I think we should be doing checking for required values on all input.

ahmaddani’s picture

Where can i get the core of aws_ec2?

All link at this page is patch..

Please... how i get it..

dragonwize’s picture

@ahmaddani

It is in CVS. This module is still under development so there are no usable releases yet.

mdayton’s picture

This discussion thread ended abruptly about a year ago. Has there been further development of this module? We are interested in a D6 module that we can use to provision and manage multiple EC2 AMI/instances.

dragonwize’s picture

I have not had any more time in the last year as a lot of other things have happened. However, I am hopeful that I will be getting back to work on it in the next couple months.

mdayton’s picture

Thanks for update Dragonwize. Am I correct to assume that as long as module is in CVS that there is nothing we can test/evaluate?

dragonwize’s picture

No, everything in drupal.org's CVS repo is open to anonymous users for reading. You can download it, test it, and help out by submitting patches if you like. The fact that it is only in CVS just indicates that the code is in early development.

davisben’s picture

Version: » 6.x-1.x-dev
Assigned: » Unassigned
Issue summary: View changes
Status: Needs work » Closed (outdated)