Talk:PERI Survey

From PERI

Pat Worley said:

I still don't like question 2a. The options are kind of random, and I am not sure if it is important to ask this question at this stage.

I'd like to see sections 2 and 3 switched, and section 3 called something different - Performance Issues?

How about something like:

1. Project Contact Information
2. Performance Issues (possibly restructured because haven't asked about code characteristics yet, and interested at highest level)
3. How many component codes: ...
   Then for each component code, ask about contact information, code characteristics, performance evaluation, so closer to current design.

---

DanGunter said:

I agree with Bronis (Note: I meant "Pat"!) that section 3 should come first.

nitpick #1: you need more contrast between the background and the white boxes. at least on my screen, it is hard to even see the boxes.

nitpick #2: are all responses required? IMHO, you are more likely to get a good/quick response if you can put a little red star next to the things that are essential. for example, "web site URL" in (1) would be nice, but is not going to invalidate the rest of the answers.

For 2a, how about just a free-form entry with the same instructions. I would add a second part, which is "Which of these solution methods are known performance bottlenecks"?

2b. seems like overkill. I would replace 2b. with a bunch of checkboxes (here I think we can cover them quite thoroughly) and a "comments" box. Anyone who really wants to evangelize, e.g., Python can do it there.

2e. Should say "On average, ..." and you should add "Not released" to the options.

2f. not all the options, ie "modified significantly" and 'ported to another architecture' are mutually exclusive. Maybe checkboxes instead of a drop-down?

3b. Typo, "What is you" should be "What is your".

nitpick #3: When I enter lot of text (i.e. in 3b) I can barely see and can't use the scroll bar

3c. Typo, trailing "yes - definite plan'

3e. Typo. Second checkbox, "determineing"

3f. There should be some more explanatory text here about what would be expected. Otherwise I don't think people will know how to answer this.

---

Phil Roth's comments and suggestions:

• Intro: would like to see statement about requiring access to full source code to be extended to suggest we will need input files for both small benchmarks and for production-sized runs. Also, ask for responses to reflect how code is used in production mode whenever it matters.

• Intro: sentence about requiring access to CVS conflicts with sentiment expressed at kick-off meeting about not wanting to work with the "bleeding edge" code.

• Intro: 'updating when convenient.' -> '...can be updated at any time, e.g. when more complete information becomes available.'

• Intro: how about adding a statement saying how long survey should take? (E.g., 'This survey should take no more than 30 minutes to complete for a person familiar with your application code base.'

• Section 2: Include introductory statement with wording that allows for the "I don't know what my current performance is" scenario. Some groups won't on the target platforms.

• Section 2: Allow responses to be expressed in either "computer science" metrics or "science based metrics." If the latter, ask if they know how the two relate.

• Section 2, question d. 'e..g' -> 'e.g.'

• Section 3: 'Other iterative methods...' -> 'Other solution methods (please specify)'

• Section 3: 'Which of these...' sounds like a question about all methods in the list. Change to 'Which of your solution methods...'

• Section 3, question b: Python example is good in that it might cause some people to think about all languages they use, but I'm concerned some people will dismiss our offers for help as "just another group of CS folks without a clue about *real* applications." I'd like to see a Fortran example line added to the Python example line.

• Section 3, last question: The question about checkpoint/restart is overly specific. If the survey is to ask about I/O, it should be several questions, for example:

   1. What kinds of I/O does your app do?
     [ ] reads significant-sized data files at program initialization
     [ ] periodically writes checkpoint files
     [ ] periodically writes results (e.g., movie frames)
     [ ] writes significant-sized results files at program termination
     [ ] other (with a box asking for specifics)
   2. For each box you checked in question 1, how large are the files, and how many are there?
   3. For each box you checked in question 1, what type of I/O is it (serial or parallel)?  If parallel, what mechanism is used (e.g., MPI-IO, parallel netCDF, home-grown)?
   4. Do you believe I/O to be a bottleneck currently?  If not, do you expect it to become one as you bring your code to your desired level of performance on your desired platform?