Fowler’s DSL example with MGrammar (Draft!)

I drafted Fowlers state-machine example with MGrammar. While doing so I found a few things I would like to see optimized or find better solutions for.

  • New line handling. I would like to express one required new line, when it is required, but then ignore all the others. Forum discussion.
    I could make new lines part of some tokens and then just interleave other tokens. But I don’t think this would be a nice solution. Any ideas?
  • The blocks events, resetEvents, commands and state+ have to be written in order. It is possible to describe a grammar where unordered blocks would be allowed, but it’s not trivial.
    Ordered and unordered “Lists” of syntaxes, separated somehow should IMHO be supported directly: forum discussion.
  • I wrote the Grammar using Intellipad and the three-pane input/grammar/AST view. It’s great! But I’d like to define the Semantic Model (using MSchema) along with the grammar and directly validate my AST against it: forum discussion.
  • References are not directly supported by MGrammar/MSchema. Forum discussion.
  • Xtext, which is quite comparable to MGrammar has a much more compact syntax. But I haven’t done a more deep comparison yet.
    Sven Efftinge’s Blog: Fowler’s DSL example with Xtext

Still missing:

  • A schema that represents the structure of the AST and validates it using constraints.
  • An adapter to Simple State Machine on CodePlex or a generating a state machine framework using T4.

Would love your feedback…

Grammar definition

module MGrammar.Net.Sample {

  // This is my first try, to parse the state machine
  // dsl syntax proposed by Martin Fowler
  language Statemachine 
  {
    interleave Ignore = " ";
    
    syntax Main = 
        NewLines? e:Block(Events, NameAndCode)
        NewLines+ r:Block(ResetEvents, Name)
        NewLines+ c:Block(Commands, NameAndCode)
        NewLines+ s:List(StateBlock)
        NewLines?
        => Statemachine 
        {
          Events { valuesof(e) },
          ResetEvents { valuesof(r) },
          Commands { valuesof(c) },
          valuesof(s)
        };
        
    // list of syntax "Item" separated by one or more
    // new lines
    syntax List(Item) = n:Item => {n}
        | list:List(Item) NewLines+ n:Item 
            => {valuesof(list), n};
    
    // lists of tokens separated by whitespaces 
    syntax TokenList(Item) = n:Item => {n}
        | list:List(Item) n:Item 
            => {valuesof(list), n};

    // a block consisting of a keyword, new-line-
    // separated "Item"s and end
    syntax Block(Keyword, Item) = Keyword 
            NewLines items:List(Item) 
            NewLines End => {valuesof(items)};
            
    // a word projected as a name
    // grouped into a separate successor
    // for extensibility reasons
    syntax Name = n:Word
        => { Name {n} };
        
    // a word plus a code, separated by spaces
    syntax NameAndCode = n:Word c:Word
        => {Name {n}, Code{c}};
    
    
    // special block and subsyntaxes for states
    syntax StateBlock = State n:Word 
            actions:(NewLines a:Actions => a)?
            NewLines+ items:List(Transition) 
            NewLines+ End 
            => State 
            {
                Name { n }, 
                actions, 
                Transitions { valuesof(items) } 
            };

    syntax Transition = e:Word "=>" s:Word
        => { Event {e}, ToState{s} };
        
    syntax Actions = tActions "{" l:TokenList(Word) "}" 
        => Actions [ valuesof(l) ];
    
    
    // keyword specification for strong formatting
    @{Classification["Keyword"]} final token Events = "events";
    @{Classification["Keyword"]} final token Commands = "commands";
    @{Classification["Keyword"]} final token ResetEvents = "resetEvents";
    @{Classification["Keyword"]} final token State = "state";
    @{Classification["Keyword"]} final token End = "end";
    @{Classification["Keyword"]} final token tActions = "actions";
    
    
    // Some whitespace characters
    token NewLineCharacter 
            = 'u000A'  // New Line
            | 'u000D'  // Carriage Return
            | 'u0085'  // Next Line
            | 'u2028'  // Line Separator
            | 'u2029'; // Paragraph Separator
            
    token NewLines = NewLineCharacter#2..;
         
    token Letter = 'a'..'z' | 'A'..'Z';
    token Digit = '0'..'9';
    token Word = (Letter (Letter | Digit)*);
  }
}

Input Text

events
  doorClosed  D1CL
  drawOpened  D2OP
  lightOn     L1ON
  doorOpened  D1OP
  panelClosed PNCL
end

resetEvents
  doorOpened
end

commands
  unlockPanel PNUL
  lockPanel   PNLK
  lockDoor    D1LK
  unlockDoor  D1UL 
end

state idle
  actions {unlockDoor lockPanel}
  doorClosed => active
end

state active
  drawOpened => waitingForLight
  lightOn    => waitingForDraw
end

state waitingForLight
  lightOn => unlockedPanel
end

state waitingForDraw
  drawOpened => unlockedPanel
end

state unlockedPanel
  actions {unlockPanel lockDoor}
  panelClosed => idle
end

Output AST

Statemachine{
  Events{
    {
      Name{"doorClosed"},
      Code{"D1CL"}
    },
    {
      Name{"drawOpened"},
      Code{"D2OP"}
    },
    {
      Name{"lightOn"},
      Code{"L1ON"}
    },
    {
      Name{"doorOpened"},
      Code{"D1OP"}
    },
    {
      Name{"panelClosed"},
      Code{"PNCL"}
    }
  },
  ResetEvents{
    {
      Name{"doorOpened"}
    }
  },
  Commands{
    {
      Name{"unlockPanel"},
      Code{"PNUL"}
    },
    {
      Name{"lockPanel"},
      Code{"PNLK"}
    },
    {
      Name{"lockDoor"},
      Code{"D1LK"}
    },
    {
      Name{"unlockDoor"},
      Code{"D1UL"}
    }
  },
  State{
    Name{"idle"},
    Actions[
      "unlockDoor",
      "lockPanel"
    ],
    Transitions{
      {
        Event{"doorClosed"},
        ToState{"active"}
      }
    }
  },
  State{
    Name{"active"},
    Transitions{
      {
        Event{"drawOpened"},
        ToState{"waitingForLight"}
      },
      {
        Event{"lightOn"},
        ToState{"waitingForDraw"}
      }
    }
  },
  State{
    Name{"waitingForLight"},
    Transitions{
      {
        Event{"lightOn"},
        ToState{"unlockedPanel"}
      }
    }
  },
  State{
    Name{"waitingForDraw"},
    Transitions{
      {
        Event{"drawOpened"},
        ToState{"unlockedPanel"}
      }
    }
  },
  State{
    Name{"unlockedPanel"},
    Actions[
      "unlockPanel",
      "lockDoor"
    ],
    Transitions{
      {
        Event{"panelClosed"},
        ToState{"idle"}
      }
    }
  }
}
Advertisement

Open Letter to Douglas Purdy: Eclipse, Oslo, and how to invent the future together

(Find his answer here)

Dear Douglas Purdy,

Over the last several months I followed the development around “Oslo”. I couldn’t attend the PDC, but I watched all Oslo videos, read a lot of the reactions, and also spent some hands-on time writing a few schemas and grammars.

The subject I want to write to you about is particularly your talk A Lap around “Oslo” at PDC as well as your post What is Oslo?.

I’ll just quote some of your spoken and written words and put my comments below. Just for easier referencing I’ll group them in sections.

Feel free to answer on your blog in order to avoid endless comments. I suggest using the tag “Oslo2EMF”.

1. Preamble

I’ve been developing using Microsoft technologies for 7+ years now. I’m absolutely positive to Microsoft and I love .NET. Nothing here is intended to be disparaging about Microsoft.

We have realized that we live in a model-driven world.

Compared to the Java and Eclipse camps, I’d say, you realized that quite late, but still welcome and congratulations. Talking about models and particularly model-driven software development, they seem to be far ahead in both methodology and technology (more later). In a way MS also acknowledged this, by joining OMG [1]* after kind of criticizing it for a couple of years.

* see link index below for all following links

2. Terms

Now when we talk about models, and one of the things we talk about particularly with modeling inside of Microsoft, particularly with Oslo, is this notion of a very key term that I like to get across, which is Model-driven Development, Model-driven Software Development.

If you want to upset the community, just take an established term “B”, rename it to “A”, and then reinvent something slightly different branding it “B”.

My Point is that MDSD is an established term in the community. It has already been defined and there are several books available [2].

The term MDSD – as the community uses it – has its roots back in the OOPSLA in 2003 [3]. Many of the attendees did MDSD before that, but they didn’t have a term that wasn’t  trademarked (as MDA, MDD are). – Prove me wrong!

“This” MDSD stands for using models in the software creation and building process. In most cases this is done using code generation frameworks such as oAW[4] or androMDA[5].

In the Microsoft world MDSD was almost unheard [6], but Software Factories and the DSL Tools, as well as some Visual Studio built-in designers could be grouped into that category.

Now, in your talk, you renamed that approach to “model-assisted”. I think this is an appropriate term and I also understand your next point about “model-driven” in the sense of models directly executed by runtimes. But in that sense, “driven” means to drive applications and runtimes, not the development process.

To make my point: Please don’t call executing models MDSD. I think Model-driven Applications, as you also call it sometimes and as written on the slides (14:50), too, is more adequate.

3. Runtimes vs. Code Generation

To stick to your terminology, you said model-assisted development is a great way of creating software today. I do totally agree with that.

And you see some great examples on that on the slide. One of the canonical ones is HTML.

You are right. HTML is a model which is driven by several runtimes and it’s even openly specified. But it’s also an great example of how hard it is to write runtimes – All browser vendors struggle – I guess, I don’t have to say more here.

Another “great” example is the Windows Workflow Foundation. I like it. I even gave talks on it. Microsoft did well, but not good enough. They have to rewrite the runtime, the tools, …

Textual DSLs – I tell you this thing is hot!

Exactly! I totally agree, and I love MGrammar.

Q: “10x more productive with Oslo than with their current set of tools” to do what?
A: To write an .net application.

We are giving you mservice, mweb, etc.

You can, of course, write your own DSL, but we think that that will be the province of mainly ISVs and people that write frameworks today.

I totally disagree. Do you really mean what you’re saying?

DSLs are broadly used in projects, not only frameworks. You’ll miss quite a bunch of (growing) audience, if you focus only on ISVs. Big companies do big projects. And there is written millions of lines of code in such projects. MDSD, already today, boosts productivity in such projects – not 10x, but still a lot. [9]

I don’t say, runtimes are bad. But it is just not realistic to create runtimes for everything you want to base on models.

One of the reasons code-generation has so bad reputation – especially in the MS world – is because it’s often too ugly. When I talk about code generation, I mean generating well-structured, nice formatted high-quality code.

For now, Microsoft’s only answer to code generation is T4. Nice, but usually it leads to ugly code. It doesn’t even support creating multiple files for different classes.

What do you think? Do you have any plans on leveraging a code generation framework?

4. Open Microsoft and the Eclipse Modeling Project

Early, open and often. This is Pre-Pre-Pre Alpha, its raw. We want to get your feedback.

Love it!

M will be released under the Microsoft Open Specification Promise(OSP)

Does it mean, the source code will be published as a reference implementation, too? And what is M? MGrammar, MGraph, MShema, MWeb, MService, …? Will the mapping to the repository, and the definition of the repository be opened?

Anyone can implement it. We want this approach wide spread on a variety of different platforms. We want it to be as broad as Xml is today.

Quite ambitious. Where is the standards consortium, then? What about JSON?

We want to engage, particularly with the Open Source Community, in order to make sure that we can invent the future together.

I’m quite interested in what that engagement could be like, when it comes to investing effort or money.

Are we going to do anything to build a bridge to EMF, ehm, the answer is: …. ehm, we will see.

The key thing is: What I’d love to see is, I’d love to see the Eclipse Model Project, I’d love to see that community to come up with a bridge for this. So that’s part of the open – I’m not kidding.

So if they want to develop a bridge for this, absolutely I’m going to support doing that. That’s one of the key reasons we do the OSP.

I’d love to see an Eclipse “M” Project. I’d love it.

Could somebody quote me on that, so we can get that ball rolling?

Quoted. So now lets get the ball rolling. We’ve got two points here. Who is “that community”, and what kind of “bridge” do you mean?

I don’t think, Eclipse just throws their well tested products away in order to use “your” pre-pre-pre-alpha code. Asking Eclipse to implement “M”, at the first glance, sounds arrogant and kind of ridiculous to me.

But, being open, lets imagine EMF was a place and we want a nice bridge to and from Oslo.

Of course there will be the feeling of competition…
But in the end, the success of modeling is the most important thing.

Ed Merks, EMF project co-lead

Where to build the bridge? And what to archive with it?

  1. EMF (Ecore) <-> MGraph/MSchema
    If you understand both models on both sides, you’ve won alot. But mapping the schemas to Ecore could be a hard issue. EMF uses explicit inheritance, while MSchema uses structural subtyping. But still, it could be done.
  2. xtext <-> MGrammar
    Both languages transform unicode to semantic models. The concepts match, while the syntax is quite different.
  3. TBD, T4? <-> openArchitectureWare
    When it comes to code generation, M2M transformations and model checks, as said before Microsoft doesn’t offer anything even approaching openArchitectureWare.

I think the people you should get involved with are Ed Merks (EMF co-lead and Ecore-developer) as well as Sven Efftinge(xtext and oAW Project lead),  Markus Völter (xtext, oAW) and Peter Friese (xtext, xtend, oAW). [7]

Could you imagine an open space conference on this collaboration? You say “I’d love it” so often that I have to ask, how much would you love it?

Sincerely yours,

Lars Corneliussen

Links Index

Since this is a letter, I didn’t want to confuse it with links. So I indexed them and put them below.

What "Oslo" is and is not (November 2008)

See also: Open Letter to Douglas Purdy: Eclipse, Oslo, and how to invent the future together

The PDC is over, and most contents including the Oslo SDK (without Quadrant) and all PDC Session Videos on channel9. Thanks! But where are we now? If you want a “brief” overview, you should at least have a look at A Lap around “Oslo”.

Jump to “What Oslo definitely is!”

Oslo Resources

What Oslo is not!

I’ll start with correcting some of my former assumptions in DSL Tools, T4, the Software Factories and “Facts” Rumors about Oslo.

Built upon DSL Tools?

No, Oslo (M and Quadrant) is not built upon DSL Tools, T4 and the UML Tooling in Visual Studio 2010. It’s totally apart. The DSL Tools are the here and now, while Oslo still is “Pre-Pre-Pre-Alpha” (Douglas Purdy).

Douglas promised that his team is working tight with the VSTX team in order to bring this together. He mentioned the possibility of using M as the backing metamodel format for the DSL Toolkit. Graphically edited DSLs could then be converted, described in MSchema and stored into the Oslo Repository or MGraph files. Update (12.Nov): Later in his talk “a lap around Oslo” Doug also said his the team is discussing to schematize the DSL Toolkit, and let the graphical DSLs live within Oslo.

A move towards UML/OMG?

No. I don’t think so. Microsoft joined the OMG and they support some UML Diagrams in Visual Studio 2010. But so far this has not to do much with Oslo.

Update (12.Nov): Someone asked about how Oslo relates to XMI and UML. Doug’ promised that they will model the UML-Domain and let the UML Designers in Visual Studio live within Oslo.

Talking of Model-driven Architecture (MDA) it seems that Microsoft avoids this abbreviation. They rather talk about Model-driven Development and Model-driven Applications – I’ll write more on this in a separate post.

A Application/Process Server?

Dublin is announced to be Model-driven, but it is not a part of “Oslo”, but rather related to Windows Azure and IIS. The plan is to host cloud-enabled services both locally on Dublin and in the cloud on Windows Azure with maximal control and minimal effort for deployment and configuration.

Read Workflows, Services, and Models by David Chappell to see how Dublin relates to Oslo.

A new Version of WF and WCF?

WCF and WF were model-driven right from the beginning, but in the next versions the source for the models that drive Workflows and Services might come from Oslo.

MService is a textual DSL that combines service configuration, implementation and workflow definition into one resource.

What Oslo definitely is

A new Modeling Language Family

Yes, it’s definitely about a new modeling language family “M”, consisting of three core languages. These languages do all have similar Syntax and a shared type system.

  • MSchema
    Defines schemas for data instances. Quite comparable to XSD and DTD.
  • MGraph
    JSON-like format for capturing concrete data instances.
  • MGrammar
    Defines grammars for textual DSLs. Could also be looked at as a Unicode-to-MGraph-tranformation language.

There are other languages planned for some horizontal domains as Database, Services and Web. But here you can even put some more “Pre”s before the “Alpha”.

A Language Workbench for textual DSLs!

Well, I’ll just copy Martin on that.

At some point I really need to rethink what I consider the defining elements of a Language Workbench to be. For the moment let’s just say that Xtext and Oslo feel like Language Workbenches and until I revisit the definition I’ll treat them as such.

Martin Fowler Bliki: Oslo

A new Storage Repository!?

Oslo Repository is a tool and a framework that allows storing models into SQL Server.  I’m still quite torn on this idea.

  • Design-time story: Many tool vendors tried backing repositories supporting simultaneous changes. But sharing them in files is just easier. If it was on the web – ok, then we had to talk about security and stuff – But as it is tied to SQL Server 2008, it looses a lot of my attention.
  • Runtime story: Having models in a database at runtime at least seams more natural in my opinion. What then to store there is another thing to figure out. Is it meant to store operational data or configurations and application definitions? That’s what is left to the customer to decide.

The repository compiler (mx.exe) generates T-SQL code out of your models, while it supports two modes:

  • Plain T-SQL: Mapps your explicit MSchema definitions or implicitely “guessed” schemas from MGraph data to data tables.
  • Repository SQL: A Repository is a database pre-populated with tons of schemas describing the “Microsoft IT world”. It also supports some core services, as are:
    • Deployment: Import and Export from and to M + Webservice Deployment to Dublin
    • Security: Tables hidden behind views, claims-based security
    • Catalog: Keeps the semantics you have in your MSchema as not everything maps directly to T-SQL.
    • Versioning: Sadly not instance data versioning. But you got full SQL Server 2008 capabilities including SSIS, Change Tracking, Replication, … + Due to the separation of views and tables you can support backward-compatible schema evolution.

Oslo does not come with a own data-access technology stack. It rather gives you a generator for EF-Models.

A Graphical Modeling Toolkit!

Yes. Quadrant lets you interact with models graphically. It’s highly generic, customizable and it looks great.

But as I’ve understand, Quadrant only works on models stored in the Repository, hence SQL Server. Really interested to see how this works together with your textual definitions you might rather store in Version Control.

Release Dates, Packaging and Pricing

From Microsoft “Oslo” Frequently Asked Questions

Q: What is the timeline for shipping “Oslo”?

A:  We are not disclosing the release schedule at this time. We are committed to releasing regular Community Technology Previews (CTPs) after PDC and will engage with the developer community on an on-going basis. The Oslo Development Center (http://msdn.microsoft.com/oslo) will also be the place to learn about the latest Oslo-related downloads.

Q: Is “Oslo” shipping with the rest of the Visual Studio 2010 product line?

A: We are not currently disclosing schedules or packaging.

Einführung in die Windows Workflow Foundation (V 3.5)

Auf der NRW08 hatte ich eine Präsentation zur Windows Workflow Foundation gehalten – zugegeben etwas verspätet ist diese nun online verfügbar.

In diesem Talk sind die Änderungen, die für WF 4 zu erwarten sind nicht berücksichtigt. Die Arbeitsweise in WF 4 bleibt die gleiche, laut Ankündigung sollten aber signifikante Probleme behoben und, was ich besonders bemerkenswert finde, das Erstellen von eigenen Aktivitäten sowie die Integration mit WCF deutlich vereinfacht werden sowie.