December 16, 2005
In the closeout session this afternoon, the DANSE Site Visit Committee made a strong recommendation that the DANSE project should be funded by the NSF. Finalization of the arrangements are up to the NSF, but the committee report is expected to be quite positive.
August 11, 2005
I think you know about the software project to build a data analysis system for neutron scattering at the SNS, Distributed Data Analaysis for Neutron Scattering Experiments (DANSE). A number of you have asked about what happened to the DANSE Construction proposal, submitted to the NSF in mid-January. The story is a bit complicated, but generally the news is encouraging.
The Construction proposal received very strong written reviews, half excellent and half very good. There were no questions about the intellectual merit or technical approach -- all referees were pleased with both. The concerns were related to the project management and the interface between the DANSE project and the SNS.
After some further discussion, the NSF has offered us the opportunity for a site visit this year, and we have accepted it. This is likely to occur in November, 2005. Within a few weeks of this site visit, an up or down decision is expected on the funding of a 5 year construction project for DANSE.
The present design award for DANSE ended at the start of August. We still have some carry-over funds in the project, and interim funds are being arranged from the DOE through the ARCS project at Caltech. This is a bit complicated to explain, but it seems to be a workable method for keeping out team together in the interim.
Technical progress on DANSE is accelerating. Feel free to try our first release for ARCS software, although it is an alpha release intended for internal use: http://www.cacr.caltech.edu/projects/ARCS/Software.html
This ARCS software is for inelastic neutron scattering, but it is built as a prototype of a DANSE system. The software is modular with fine granularity, and comprises over 4.000 individual files. It is a technical challenge to manage the release of software based on all these different files and their different licenses, and keep it simple for a user. Tasks such as these have moved us much further into the realm of a software engineering group than I would have thought. Science still drives the effort, of course, and you can see this on the DANSE MediaWiki.
May 20, 2005
We are pleased to announce that the arcs and danse cvs repositories are now available through anonymous access. Since the arcs repository has three years of solid use behind it, and the ARCS alpha release is fast approaching, it is an excellent time to reorganize the repository.
The major modifications to the repository have been completed, and the anonymous access has been activated. These changes required moving the repository off of it's old home at arcs.cacr.caltech.edu to a new home on the standard CACR cvs server. We are in the process of updating the corresponding documentation on the DANSE wiki, so bear with us for any inconsistencies in documentation in the short-term.
-- MAPPED STRUCTURE OF ARCS AND DANSE CVS --
We are working on a better way for the user to get a better sense of what exactly the structure of the cvs repositories are. Currently, the directory structure for ARCS and DANSE will be displayed at http://wiki.danse.us/danse/index.php/DANSE_cvs
Also included is a brief description of the contents of each package. All major changes to the repository structure are reflected in the documentation on this wiki page, and this will be updated as the wiki structure continues to evolve.
-- UPDATED LIST OF DANSE/ARCS PUBLIC RESOURCES --
- DANSE-related questions -> danse @ cacr.caltech.edu
- ARCS-related questions -> arcs @ cacr.caltech.edu
- PYRE-related questions -> pyre @ cacr.caltech.edu
- CONFIG-related questions -> config @ cacr.caltech.edu
- DANSE homepage -> http://wiki.danse.us/danse/
- ARCS homepage -> http://www.cacr.caltech.edu/projects/ARCS/
- PYRE homepage -> http://www.cacr.caltech.edu/projects/pyre/
anonymous access to the CONFIG build procedure
cvs -d :pserver:firstname.lastname@example.org:/config login [password: config] cvs -d :pserver:email@example.com:/config co config
anonymous access to the PYRE framework
cvs -d :pserver:firstname.lastname@example.org:/pyre login [password: pyre] cvs -d :pserver:email@example.com:/pyre co pythia-0.8 cvs -d :pserver:firstname.lastname@example.org:/pyre co templates
anonymous access to DANSE components
cvs -d :pserver:email@example.com:/danse login [password: danse] cvs -d :pserver:firstname.lastname@example.org:/danse co danse
anonymous access to ARCS components
cvs -d :pserver:email@example.com:/arcs login [password: arcs] cvs -d :pserver:firstname.lastname@example.org:/arcs co arcs
- Request a wiki account
- Install and build instructions
- Map of DANSE/ARCS cvs repositories
- Weekly meeting notes
We hope that you will have a continued interest in the ARCS and DANSE project, and will continue to track our progress and try our software. If you have any other questions or concerns, then please feel free to contact either myself or Brent Fultz.
--Mike McKerns 16:22, 20 May 2005 (PDT)
February 9, 2005
Here is additional information about the neutron hardware and software meetings at Caltech in the week before the APS March Meeting. There are three meetings in sequence. The first two are on software development; the third is on ARCS and SEQUOIA.
1. March 16, 17, 18 AM -- tutorials for postdocs and interested persons (such as me). Morning instruction and afternoon working sessions are planned. The level of instruction will be set to benefit the most attendees. A fairly-complete list of skills needed for being an efficient developer of DANSE software is given below. The expectation today is that the level of instruction in March may be set for persons with a good knowledge of the Primary Skills. Does this work for you? Please look over this list and give us some more detailed feedback on what you would find useful.
2. March 18 PM, 19 -- DANSE developers' meeting The focus will be on planning our work on the DANSE project for the next few months. These are listed in the DANSE Project Plan, available upon request. Some topics are given at: Project_Management
3. March 20 -- ARCS/SEQUOIA IDT meeting Project updates, user needs for sample access, magnet issues, single crystal experiments.
Please email Polly Thompson (email@example.com) now to let her know your travel intentions. Polly needs to get started in arranging a block of hotel rooms, and your travel. We also need to arrange offices at Caltech for your postdocs to work on March 16-20.
January 24, 2005
The full proposal with Project Plan, as printed from the NSF Fastlane web server, is 362 pages and 15.1 MB. I cannot send it by email. You should be able to get the Excel Plan from the DANSE Restricted MediaWiki. The Plan and the attached proposal gives you all technical content (without c.v.'s, financial information, and miscellaneous legal documents).
With help from Tim, Michael, and me, Mike managed to fill in all the missing level 5 detail in the Excel spreadsheets. All of the annual subproject budgets of course agree exactly to the annual totals in the spreadsheets, and most of the subprojects agree to within $ 1 in the allocations of labor and fixed costs. Deviant ones are off by only a couple of k$. Out of M$ 13.8, this is quite remarkable, in my opinion, and is certainly good enough for a project plan.
As we discussed in August, it wasn't obvious that we could do this properly by Jan. 24. It was of course crucial that all of you were so responsive. I appreciate that many requests and actions did not make sense at the time, so I thank you for your tolerance too. I know that a couple of you were rather annoyed with how your budgets were derived, and thanks for putting up with us. Remember, though, the budget details await a real budget. I hope it will be closer to M$ 13.8 than to $ 0, but even these extremes are possible.
September 16, 2004
The workshop next week will start as scheduled; however, there is still a possibility for changes, especially in the content and timing of the talks. The focus clearly has to be on organizing the large DANSE proposal, which will take enormous effort to complete before the end of November.
We also have prepared some sketches for the DANSE proposal. Please read the abstract with care. The abstract points out what you might find interesting in the document, even if you do not have time to read carefully all the contents. The sections on Common Features of Scientific Subprojects and my own section (with Frans Trouw) on Inelastic Scattering might prove useful as a template for the organization into a Work Breakdown Structure. You will receive this document and the agenda in a black packet that you can pick up during check-in at your hotels in Pasadena.
Some maps for ground transportation are at: http://admissions.caltech.edu/visiting/maps
Thanks again for agreeing to come to this Workshop, and I look forward to seeing you on Monday morning. We have a lot to do in a short time. I hope we can make some good progress at this Workshop.
August 31, 2004
We will be holding a DANSE Workshop at Caltech on Sept. 20 and 21, a Monday and Tuesday. We will start in the morning of Sept. 20, and there should be a close-out session in the late afternoon of Sept. 21.
The purpose of this workshop is to prioritize the tasks for the DANSE Project. My latest information from the NSF is that a construction proposal will be due on Dec. 16. We will therefore follow the ambitious fast-track schedule that we discussed last summer. This is our one meeting to plan the DANSE project.
An agenda should be available in a week or so. Approximately it will include presentations by subproject leaders about their experiences with the DANSE framework and community polling, presentations on the prioritization of tasks in the science and software engineering efforts, and of course the ideas and critical discussion that we need from all if you.
If you are interested in attending, please contact both Pam Albertson (mailto:firstname.lastname@example.org) and Robin Hanan (mailto:email@example.com) about your travel arrangements. Pam and Robin can work with you to get suitable airfares, ground transportation, and hotel arrangements. There is some money available for travel reimbursement for those of you not funded directly by DANSE. Please contact me (mailto:firstname.lastname@example.org) for further information.
I hope to see you at Caltech.
August 25, 2004
I have been most impressed with your quick progress in developing software components. Some of you have even completed the polling of your community. Framework development is also moving along well. In the next month we should have a solid proposal for the content of the data streams.
We now need to schedule a meeting to plan the scope of the DANSE project in more detail. This amounts to organizing the tasks for the construction proposal. With luck this will be the only meeting needed until the spring of next year, or late winter anyhow. Invited to the meeting will be a specialist from Triad Project Management Services who will work with us as a consultant to help prepare the work breakdown structure of the construction proposal.
I need to start collecting dates when you are unavailable to come to Caltech, but unfortunately Elizabeth Welsh has just left us for a new life on the East Coast, and we have not yet hired her replacement. I need some help from you to schedule the workshop.
If you are interested in attending, please go to our calendar on the web and mark the days you are not available in September and October. From your web browser:
- login as:
- Select the month of September 2004 (at bottom). (drag your mouse over the entries to see how they display)
- Mark the dates that you are not available in September and early October. To do this you need to:
- click on the upper right button for the day in question (a plus sign)
- enter your initials in the "brief description"
- enter "unavailable" or other information in the "full description"
- (follow the style of others if you like)
- After an event is entered, you can edit it, but this is not so obvious. First, click on the event. In a new page, towards the lower left of your browser window, you will see:
they are not prominent, but they work.
Your help with this will really help me organize the meeting. In the past we have had a lot of ambiguous emails and changes of availability that took real effort to get straight.
Incidentally, this is a temporary arrangement with the calendar, and we will likely change access in a few weeks. I hope to see your responses soon.
August 6, 2004
Thanks to the efforts of June Kim, Mike McKerns and others, the editable DANSE web site is up an running. Please bookmark it.
The DANSE site is a MediaWiki, related to the open source encyclopedia that has been featured recently in the newspapers: http://en.wikipedia.org/wiki/Main_Page
The MediaWiki evolved from the group that provided the core of the ARCS wiki, but the MediaWiki is a real advance. It is well-suited for evolution into a type of on-line documentation, since it is fairly easy to rearrange and manage content.
Editing requires that you log in to the site, however. Please request an account as explained on the first page of of the DANSE MediaWiki. Notice from the topics bar on the left that there are a lot of categories that are ready for your input.
Take a quick look at these pages on the MediaWiki:
July 27, 2004
By now most of you have sent a ssh public key to email@example.com, and therefore you can use our repository at arcs.cacr.caltech.edu
Instructions and a guide on how to interact with the repository are now available.
For those who do not yet have accounts on our repository, we have detailed instructions on how to generate your public key. Further, detailed instructions on how to install the Pyre framework are also available.
There has also been good progress from some of the subgroups on building a first pyre component for the DANSE framework. We have started a cvs directory for those of you that have cvs write-access and want to deposit your work-in-progress code (the new directory is named "subgroups") in our repository. Depositing your code in our repository will make the interactive process between the subgroup and the Caltech DANSE developers run much more smoothly and quickly. Instructions on how to deposit your components in our repository can be found here.
For those of you that have installed the build procedure and framework on MS Windows (using Cygwin), there is an important update. "config", "pythia-0.6", and "templates" are now more flexible and functional on Cygwin. Please update your installation.
-- Mike McKerns
July 22, 2004
A couple of days ago I received the message below from the NSF. They were happy with the response to the reviewers of the Reverse Site Visit, and the award for DANSE, DMR-0412074, is going forward.
I had a promising meeting with a Richard Fischer of Triad Project Management Services, Inc., the organization that supports the NSF-funded LIGO project, for example. They are interested in helping with our development of the WBS for DANSE, and perhaps providing long-term project controls service and help with earned value reporting. They have exactly the talent that we need, but they are not cheap. This may be a sub-contract of k$ 30-50. It is a necessary cost of doing business, however, since the credibility of the WBS is crucial for the success of the construction proposal.
I am pleased that so many of you are underway with the component development, and this is generally going well. Note the attached schedule for the development of the construction project. The components have to be in hand, or close to it, by August 13.
Also, please note that you should have polled your user communities by August 20. This date is a bit negotiable, but it needs to be done soon for the following reason. You will need to justify what your community wants in order to prioritize the tasks in the construction proposal. The sooner this is done, the more efficiently we can prepare the WBS. In principle, the 5 pages of text that you write for the construction proposal should be based around these prioritized tasks of the WBS. We expect the WBS to take about 3 months to get right, so we need to start on it soon.
Some of you have still not arranged for cvs access! This is an essential step if you want involvement with this software development effort. Even if you do not contribute to the development of code, access to cvs is how you will get the DANSE products, services, documentation, and it is a precise monitor of the actual work that goes on in the group.
On the technical front, the Cygwin problems for the build procedure are now solved, and further information should be forthcoming shortly from Mike McKerns. Incidentally, we are posting some help and documentation for the pyre framework in FAQ and Tutorial format.
We are setting up a new "MediaWiki" for the DANSE project. It is much more capable than the previous ARCS "swiki", and is appropriate for a project development site with sections for documentation, document control, and discussion. Some information on its capabilities is given at: http://wikipedia.sourceforge.net and http://meta.wikimedia.org/wiki/MediaWiki_User%27s_Guide
As soon as I get a firm start date from the NSF, I will request a "pre-award", which is a loan from Caltech. With this authorization, I can get started setting up your sub-contracts.
I am away on vacation all next week, but Michael Aivazis and I should be back in the first week of August. If you have any questions for me about the components or the poll, please send them by Friday morning, since I leave on Saturday. Mike McKerns should be able to arrange help for you next week, however.
July 13, 2004
There has been rapid progress on DANSE funding by the NSF. Last Thursday I received the comments from the reverse site visit, and a list of items requiring response to the NSF before Monday (yesterday).
You will probably be surprised by the response to item 2. This change is both essential for funding of DANSE, and it will make the interactions with the SNS much smoother in the future. I am optimistic that we can work together well in this new arrangement.
Note that item 3 is our schedule for submitting the Construction
proposal before the end of the year. Key dates for you are:
Code componentization by August 13
So far we have made good progress with Ersan's component (a large FEM code) and we have a clear picture on how to proceed with Simon's component (we will bite the bullet and do Python bindings to his FORTRAN pdf fit package). Paul B. and Paul K. -- I will call you shortly.
Completion of sub-community surveys by August 20
This is important for credibility when writing your five pages of the construction proposal. You need to show that your sub-project will produce something that your community needs. Even if you get only 10 responses, they will certainly be helpful. For your convenience, I have attached our old inelastic scattering survey that you can modify or ignore.
G. Tessema of the NSF has promised full funding before the end of this fiscal year, which I take to mean that funds will be available in late September. I am exploring the possibility of arranging a pre-award so you can get your sub-contracts in place in September. My intent is to give you all that you asked for (~99 k$) in spite of some other unexpected costs of doing business with the NSF.
It really looks like DANSE is now off and running.
July 7, 2004
The presentations from our recent Software Technology Workshop are now posted on the web page: http://arcs.cacr.caltech.edu/arcs/ARCS/DANSE_Workshop_June04.html We will move these to a DANSE Mediawiki once we set it up. If you have something else to contribute to this web page, such as a Powerpoint file showing a futuristic GUI, please send it to me so I can finish this web page.
There are three followup items from the Workshop listed below. These involve tasks that the subproject leaders need to start before we can really get going on a full construction proposal.
- Component development
- One of you, Cahit Aydiner, has a clear path forward for integrating a major finite element code into the Pyre framework. This is an exciting example of a major component for engineering diffraction, and it has already taught us about the practice of component development. Two components for this task are under development, both written in pure Python.
- Mike McKerns is looking forward to working with the rest of the subgroups to componentize your piece of code. Could you please reply to Mike and me about a potentially interesting component? This requires some discussion beyond a simple email. Mike will help you get started, discuss the best path for developing your code, and work with you on an iterative basis to complete the development and integration stages. Tim Kelley has also volunteered to help.
- To our knowledge, only one of you has installed both the Pyre framework and the Cobra client software on your laptop.
- By the time you receive this message, there should be a working solution for many of the problems with the Cygwin installation of the Pyre framework on Windows systems. (The installation instructions will describe the issues.) Linux and Mac installations have been reliable since before the workshop.
- The Cobra client software can be installed under Linux, Mac OS, and Cygwin/Windows. There are some tricks required to get the OpenGL working, but these are now well-documented for Cygwin/Windows. It is possible to install Cobra without OpenGL, and the only real loss is the molecular editor.
Here is the list of who was volunteered to work on the development of algorithms or capabilities of broad interest to all scientific subgroups. Please take a minute and check it, especially the items with your name on them.
June 15, 2004
What follows are descriptions of the work you will do before, during, and after the workshop:
In the course of the meeting, we want you to install the Pyre framework on your laptop computer, and set up the demo Cobra client software so that you can run the demo DANSE system. This is the best way to see your software component interoperate with the others.
Some of the components are available in our cvs software repository. For you to get cvs access to the software repository, please fill out and submit the form requesting an account on our system. The web form is at: http://bill.cacr.caltech.edu/Cacr/userform.cfm After you fill out this form, please email Michael Aivazis firstname.lastname@example.org (mailto:email@example.com) alerting him to tell the administrator that you have requested access to the system. He will request your public key, or direct you to instructions for generating it.
We strongly encourage you to look at the documentation on
the Pyre framework. It is still being revised, but good
overall explanations are available: http://www.cacr.caltech.edu/projects/pyre/docs/ch01s02.html
You will download and make:
pythia-0.6 (not 0.5. some tuneup of setup.sh)
Another site that describes Pyre is: http://arcs.cacr.caltech.edu:8000/arcs/51
Please bring with you two types of files on your laptop:
- real NeXus data files from a neutron instrument so Tim Kelley can adapt his NeXus reader to your particular instrument in a short time at the Workshop.
- a piece of code that you would like to see as a component. This is easiest if the code is in Python, but C, C++, FORTRAN (yes, even FORTRAN 90) can be made into a component with Python bindings.
We recommend that you bring a wireless ethernet card with your laptop, but there are some wired ports available in the room.
Mike McKerns is working on some installation instructions.
Finally, please tell us your laptop computer operating system and version number.
June 7, 2004
The news today is that G. Tessema of the NSF will recommend full funding for the DANSE design proposal. Funding is about as certain as it can be at this stage, although there remain some issues to work out in response to the Committee report, which I have not seen. I expect that I will have to address management issues,especially DANSE-SNS interactions.
It seems that you are able to attend a DANSE workshop on June 22 and 23. Owing to schedule conflicts, we will have a second workshop in July for the others unavailable on June 22 and 23. Unfortunately, Michael Aivazis is away for the month of July, so the June workshop is really the best opportunity for us to meet together as a group. I appreciate that this is short notice, but we have no choice if we are to complete this large construction proposal for the NSF by October. This is an enormous amount of work, and you will have to pitch in to help.
The rest of this message is a set of instructions for how to prepare for the workshop.
Please bring with you some cartoones of your dream user interface. Also, if you have collected information on the needs of your subfield community, I am sure we would all be interested in hearing how you did it and what you learned.
For the upcoming DANSE construction proposal, you will have to prepare a detailed Work Breakdown Structure for your science subprojects, using Microsoft Project. I will help you with this, of course, but the key part of this effort will be to identify the tasks, prioritize them, and estimate the effort required to execute them. Remember, your budget is exactly the total cost of the tasks listed in the WBS.
The best way to learn how to estimate the effort required for software tasks is for you to convert a piece of code into a component that will run under the DANSE framework. The actual process of doing this will be performed at the workshop. Our plan is to get your code working under the demo framework (called Cobra), but code that will work as a Cobra component will be quite adaptable later to the final DANSE framework. To be ready for this, however, you will need to do the following:
- Select a piece of code that would be interesting to interoperate with the other components in the DANSE system. Most of our components are of course for inelastic scattering, since this has been our effort under ARCS. We do have Matlab, gnuplot, and ISAW components, and a generic NeXus file reader.
- The NeXus file reader was designed for inelastic spectrometry data, but it can be quickly adapted to other types of NeXus data. Please bring some NeXus files with you. At the meeting we will configure the reader software to work with them.
- Structure your code with a clear Python interface. This means that you need to identify all required inputs, and put them in one place in the code. This can be done in two ways. First, you could write the code as in the attached file. Notice that it has a main function called "run" that puts all inputs in one place. (The type of output is also specified in the input variables.) I will be pleased to send you the actual programs including the "mpFunctions" file that is imported, and does much of the work. Second, you could take a piece of code in C, C++, or perhaps FORTRAN, and identify all input and output areas. If there are not too many entry points, we can work with your code at the workshop to provide it with Python bindings, and make it an interoperable component in the DANSE system.
- You must have Python installed on your computer. (I assume you have done this already, since you are part of the DANSE proposal!) You can install the DANSE client software at the workshop. This is easy if you have a Linux operating system. For the short time of the workshop, installations for Windows systems are only practical if you install Cygwin, a free package that gives you a Linux shell running under Windows. Please let me know if you want to take this route, and I can provide you further information. (Also, let me know if you are a Mac user.)
What you will learn from the first three steps, and see more clearly after step 4, is what it means to write software that can be used in a component framework. This exercise has been most useful for me personally, because it shows the additional time and thought required to write reusable code. Once you start thinking in terms of code components from the beginning, there is not so much extra effort. However, many codes that exist today are not at all modular, and you may need to evaluate whether they should be converted into components or rewritten from the beginning. The workshop will help show you what this means.
Also, you really need to see how the components, including yours, interoperate in the DANSE system. The interoperation of the components, which can be rearranged quickly, is fascinating. It opens up new doors to creative experimentation with data analysis, and you will see this immediately when your component is part of the system.
June 1, 2004
The Reverse Site visit at the NSF for the DANSE design proposal was as positive as we could expect. We will not know about the funding situation for about 3 or 4 more weeks, but I am optimistic.
Timing is now a critical issue. A full construction proposal will be due on Oct. 15, meaning in practice that it will have to be done before Oct. 1. This is a massive document that will require 2 months to prepare. Remember, it needs a full work breakdown structure with all tasks and effort itemized. Its budget equals the sum of the costs for all the tasks (i.e., too few tasks, too little budget). To complete the proposal on time, we need to complete the WBS by early September. I anticipate that the WBS must be started before August. This is less only 6 or 7 weeks from today!
As you remember from our attempt to do this WBS last year, it is important for you to itemize all your tasks and provide accurate estimates of the time each will require. We have the ARCS software effort for comparison, but you will need the experience of componentizing a piece of code and seeing it installed on the DANSE system as a working component. This means you will have to encapsulating some old code with a well-defined interface, or write something new. To estimate effort, there is no substitute for actually having done some work already.
Our original plan was to have a workshop at Caltech on May 26 that
would discuss the encapsulation of code, but this was postponed by its
overlap with the NSF Reverse Site Visit. We need to re-schedule this
two-day workshop, and owing to travel schedules at Caltech the best
time window is:
June 21 - July 2
This is very soon, so I need to hear about your schedule right away. Please check your calendar now and get back to me.
The purpose of this two-day workshop is essentially the same as it was stated in the previous email of March 30:
- We need to start sub-projects on code encapsulation, using a real piece of code. Our plan is to include your code contribution into the demo system that you have seen in earlier versions of DANSE so that you can see real interoperability.
- We need to get started with a discussion on high-end interactive graphics, and discuss the user interfaces. Please plan on bringing at least cartoones of your ideal user interface.
- We need to explain where we are today with the real distributed framework. Many parts of it are working, and they are interesting to see.
Even before this meeting, I hope you can get make a start on componentizing some code. I will be in touch with you later this week about how to proceed with this. In the meantime, please tell me your availability for later this month.