yas [yas]

11/09/2010: Uploaded cloud supporting modules

Hello Brian,

Thank you for your comments. I am sorry that I made mistakes. Actually I re-read at http://drupal.org/node/539608, noticed my mistakes and came back to this post, then I found your comments. Really appreciated. We are so serious that we are ready exactly to follow your (community's) any suggestions and/or comments.

Regarding Drupal coding standards, we have been so careful for that and checked through coder module before I uploaded. I believe there is no warning.

As my current concern, I wonder whether it is okay to revise and use other one's module especially like:

• http://drupal.org/project/aws
• http://drupal.org/node/214944

We built Amazon AWS and EC2 related-modules based on Drupal cvs repository which doesn't seem to be maintained. I contacted the original owner a couple of times but I don't have any response. We want to take over those.

I will upload my revised zipped archive again asap, but please wait for a while. In the meantime, I wonder if you can take a look at my DrupalCon CPH's presentation slides at: http://www.slideshare.net/yasnaoi/how-to-manage-your-cloud-by-drupal-dru...

Hello Brian,

(Revised code)

Please find the zipped attached. I believe that we've done the todo list.

(AWS module)

With following your suggestion, I am posting an offering maintainership for the modules (issues) below because we reviewed, designed, and restructured those modules by considering the extensibility.

- http://drupal.org/node/967950 (Amazon Web Services API) (Completely changed)
- http://drupal.org/node/968638 (Amazon EC2 Console) (Completely changed)
- http://drupal.org/node/967964 (REST Client) (Followed the Drupal coding standards, almost no change for logic)

I am so happy that you are interested in our project.

Thanks,
Yas Naoi

Brian,

Thank you for your support so much. I will follow the requesting scheme (-> needs to review) next time.

Best regards,
Yas Naoi

@dragonwize

I really appreciate you for taking your time to review. We are now following your comments one by one. Please let me reply as follows:

1. There are still copyrights and licences in all of the files that have to be removed prior to getting them into CVS.
   • yas: Now I am asking about this to our company's lawyer...
2. Version lines should be removed from the info files.
   • yas: Thank you for pointing it out.  We removed them internally.
3. Consider removing or changing your packages. Several of these modules are being put into packages by themselves. Refer to http://drupal.org/node/231036#package
   • yas: We are re-considering the package names which are equal to directory names as slide #1 of the attached file below.  If possible, I would like you to review it and please feel free to give your feedback.
4. Your rest_client module looks fine and I would be happy for you to take over maintainership of it as I do not have the time for it any more.
   • yas: Thank you so much.
5. Your aws module is looks like good work in the same direction as mine. I would be happy to take you on as a co-maintainer and work with you on it.
   • yas: Thank you so much.  In aws module, CamelCase is used in function names.  For current implementation, we respect your function names however we want to rename the names with underscore (_) in order to keep our design consistent (e.g. aws_queryRequest -> aws_query_request).  Is that okay?
6. Your amazon_ec2 module is not an api module but an implementation module and it is not in the same aws namespace so it should not be included in the aws module but something separate.
   • yas: That's correct.  Currently there are a few kinds of Amazon EC2 APIs compatible middleware like Eucalyptus and OpenStack (EC2 API version).  In order to support those, we separated EC2 APIs (libraries) and implementation (Please see also the slide #3 (Module Diagram) in the attached).  I thought that amazon_ec2 module would be better to be included into AWS package when we think about other AWS services like S3, etc in the future, however, as you can see the slide #1 (Directory Structure) in the attached, amazon_ec2 could be included into IaaS package.
7. There are many modules within other modules that do not keep the same namespace. All submodules in a module package should keep the same namespace by prefacing with the main modules name. This is to prevent name collisions as only the base module is in the CVS namespace and checked for uniqueness.
   • yas: I would like to review the slide #1 in the attached as well regarding the namespace.
8. Lessoning, or preferably removing altogether, the use of global constants would be good. Uses like the module name and others are polluting the global space for real reason as the module name does not change and is descriptive already.
   • yas: If our new proposal for the namespace (based on directory structure), we will match the global constants prefixes to the module name (e.g. SCRIPTING_TABLE -> CLOUD_SCRIPTING_TABLE)
9. Better basic documentation would be nice as well. It is really unclear what most of these modules actually do from reading their description and readme.
   • yas: I admit that our documentations are not enough.  We will try to put comments into source code and also to make better documentation.  Meanwhile, we will be happy if you can especially see the slide #1 and #3 in the attached.
10. You should change your style.css to module_name.css so that it does not conflict with theme styles.
    • yas: Thank you for pointing it out.  We changed the name internally.
11. Functions need a doc block with them in the Drupal format. http://drupal.org/node/1354
    • yas: Related to the item #9 above, we will try to make it.
12. hook_menus and possibly others need to be updated to Drupal 6. No $may_cache, no t(), etc.
    • yas: Thank you for pointing this out.  Historically we started to implement from D5, now we are implementing D6, (and D7 in the future), so we will update them.
13. There is definitely good work here but in many ways it is too specific to really rise as great Drupal modules. I would recommend trying to generalize your code more and remove as many dependencies on external systems like JAVA, Python, PHP CLI, etc as possible. That will make the code much more usable by a wider audience.
    • yas: Thank you for your advice.  There are some reasons for using Java, Python and PHP CLI.  I hope that the followings are helpful for your information: 
      • Python #1: We also created a plugin for an instance monitoring middleware, which called 'collectd'.  It requires to implement in Python for collected plugin...
      • Python #2: Also for supporting libvirt for server side, it only provides C, Java and Python libraries...
      • Java: We supported SSH console for Amazon EC2 instances and used MindTerm (A thirdparty Java Applet for SSH console), then we need to transfer and inject a private to key to a client, so we created a supplemental signed Java applet to do that.
      • PHP CLI: In order to realize automated operations, we are supporting script injection and execution on an instance (especially in booting time) by spawning PHP CLI apart from Apache process (e.g. http://www.php.net/manual/en/function.pcntl-fork.php#49949)  (Maybe there would be better way?)

@brianV, kiamlaluno, nowarninglabel, promisingram

Thank you for reviewing and your concern. We will make our more effort on this.

Hello community,

We have updated our source code based on the comments to the attached. At the same time, we made huge efforts for refactoring.

The major additions and changes are:

1. Additional modules
   • Cluster
   • Scripting (w/ Inputs)
   • Alerts (Currently do nothing)
   • Eucalyptus
2. Structure Change (Package and namespaces)
3. Improved UI (Especially in action icons)
4. Multiple cloud supported by Cluster module
5. (Remote) script execution at instance's booting time supported by Scripting (w/ Inputs) module
6. Some comments for Doxygen
7. Basic test cases by SimpleTest

Please let me upload the revised code and re-reply to the following comments:

1. There are still copyrights and licences in all of the files that have to be removed prior to getting them into CVS.

• yas: Our company's lawyer asked us to keep that in the header of our source code... Actually it means that we just declare to select GPL2 and include disclaimer.

• yas: Done.

• yas: We entirely re-considered the package name and module names as shown in my previous comment.

• yas: We modified a little for formatting the source. Any logic has not been changed.

• yas: We changed the function name based on CamelCase to underscore (_)-based to keep our design consistent

• yas: We put amazon_ec2, openstack_nova, and eucalyptus module into 'iaas/modules' directory.

• yas: Now we added cloud_* prefix in the sub modules under cloud package to keep the same namespace. I believe it will be able to prevent name collisions.

• yas: We changed all the sub-modules and keep the same name space like cloud_*, so I think now they are not polluting the global space.

• yas: Comments and basic documentation are still not enough, however we are trying to add some comments onto the function for Doxygen.

• yas: Done.

• yas: As I mentioned above, now we are trying to accommodate Doxygen. Actually we are starting to create a Doxygen doc w/ GraphViz.

• yas: We made so much effort on supporting D6 menu system.

• yas: Still we needed to include the PHP Cli, Java for SSH console, etc.

Thank you for reviewing our source code. We really appreciate for your feedback.

PS
If you are interested in this project, please vote for this project's session proposal for DrupalCon Chicago 2011 at http://chicago2011.drupal.org/sessions/clanavi-how-manage-your-cloud-drupal.

I talked about the license with my company's laywer again. Let me clarify the issues like nowarninglabel as follows:

1. Copyright
   As nowarninglabel mentioned, we still think that we can put a copyright notice because we need to make clear about who we are. Please refer the line from 283 to 309 in LICENSE.txt in Drupal core distribution and the other modules. In my understanding, that includes an example to describe “how to apply GPL2 to our new programs” and our source code header just intends the same thing as the example.
   In general, if a company start to talk about ‘copyright’, it may imply to get credit - a kind of DRM or something to protect a content’s right (at least I guess it...), yes that’s true, but again, in my understanding, the meaning of putting copyright notice in GPL2 seems to be a little different (or additional) context from a Hollywood company’s one. The GPL2 license requires us to include the copyright notice. Therefore, it is not redundant. Further, The Free Software Foundation lawyers do not protect our company.
   FYI, I also found even Drupal core source code includes to give a copyright header in _locale_export_po_generate function in locale.inc in order to generate .po file. So if we see .po files in Drupal, they come with a copyright header.




2. Maintaining Issue
   We don’t have any intention to quit to maintain our source code as a company immediately at this time, and rather we will eagerly continue this project; actually we are considering that to apply CVS account here is our starting point in our project. Therefore, let us change the terms in our source code header as follows:
   • “DOCOMO USA Labs may not be held liable for failing to maintain or support the software, and DOCOMO USA Labs is not responsible for any damage or loss experienced from using the software.”

I changed all our source code headers. Please find the attached again. dragonwise, nowarninglabel and others, thank you for taking your time for your concern and comments. We really appreciate for any feedback on this issue.

@jason

Thank you for your precious comment. In the meantime, we are still continuing refactoring.

Please let me upload the latest version of our source code.

• Fixing:
• Cluster functionality
• So many bugs and issues

• Added:
• Auto-refresh on instance listing view (update status automatically)
• OpenStack nova Support

Also please refer to the docs by Doxygen at http://docs.clanavi.org/

Thank you.

Now we are applying for permission to create full projects at:

http://drupal.org/node/1104978

We are continuing our development; the attached is the latest version of our effort.

• Bug Fixes
• Improved to "enable / disable" buttons
• Fixing Image Listing
• Filters Feature
• Permissions about AWS API compatible IaaS Families

• Addition / Improvements:
• Cosmetic Improvements
• 'Reboot' function for Amazon EC2 (AWS family)
• 'Snapshot' Deletion
• 'Security Group' Management
• 'Lock' Functionality for EC2 and XCP
• Configurable SSH Username
• OpenStack nova Support

Thanks,
Yas Naoi

@zzolo,

Thank you for taking care of that. I just thought that we should have continued this thread at new project application (Git) to create full projects because this status had changed into postponed.

Thanks again,
Yas Naoi

@Shadlington

Thank you for pointing it out.

We are continuing our development; the attached is the latest version of our effort (2011-04-05 ver.0.91).

• Bug Fixes:
• Fixed pagination (Listing AWS EBS Volumes)

• Addition / Improvements:
• XCP NIC Assignment by bonding consideration

@brianV
@kiamlaluno
@nowarninglabel
@dragonwize
@promisingram
@jhedstrom
@jason.fisher
@zzolo
@Shadlington
@laura s

Thank you so much for your reviewing, helping, and supporting us.

I am so glad because this project originally started from July 2009 and has passed for almost two years.

Also, I would like to thank my colleagues' huge, huge efforts for this Clanavi system development!

PS

We would like to set up our project link at /project/cloud, however it says "This project name is already in use." in promoting project page. I think the name space 'cloud' is already taken and we cannot access /project/cloud (access denied). Does anyone know the contact information to address this?

@nowarninglabel

Thank you for the information. I will ask about it through the webmaster's queue.

@Dave Reid

Thank you for your comment. I will delete 'external libraries' from cloud repository later soon. The cloud module is a set of cloud management system (We name it Clanavi), and it is somewhat big system, so I included the external modules to make it work for evaluation.

@Dave Reid

I really appreciate you. I wanted to let you know that I got the git repository cleaned up.

@jason.fisher

Thank you for your precious comment. You are correct.

@baldwinlouie

That is great, thank you for your effort! I will include that file into our cloud project.