+1

Hey Cody, thanks for sharing your concern. Apologies for the lack of transparency from our end- it's coming from a place of limited resources, not deliberate opacity. Like you said, we're a small team spread across a few large projects right now; and it's part of why we've been in more of maintenance/stability mode with Sails and Waterline over the past 9 months.

Another thing has happened too: we've become more attuned to the full lifecycle of development with a Sails app. The work we do outside of open-source has always been a huge influence on Sails-- I mean, I firmly believe the reason why the framework has helped so many other folks is that we've always focused on solving real-world problems that we have ourselves. Now that we are a product company (as opposed to being focused on services), our team has become more attuned with the experience of working w/ a Sails app over the course of months or years (as opposed to the experience of the first 2-3 months). As you might expect, maintaining our own large-scale production deployment has drawn our attention to issues like security and performance more than ever before.

What We're Up To

To set the record straight, here's a quick look at what we're currently working on:

Sails core

• Sails.js in Action (Manning book)
• Low-level work in Waterline: I'll let @particlebanana chime in to expand on anything if you have questions, but the highlights are:
  • improved SQL and noSQL query building (implemented as a machinepack)
  • a new, context-free intermediate query language to describe the operations that Waterline builds (i.e. Waterline criteria gets converted to declarative query language, which then gets transformed to the db-specific query). This detangles two very complicated processes, allowing them to be tested and developed in isolation. It also gives us a standard, declarative representation of the physical-layer queries that we send to the database; a big missing piece necessary for implementing more efficient joins on a per-database basis.
  • standardized access for underlying database drivers w/ transaction support, also implemented as machinepacks (and starting w/ Postgres)
• We're keeping our eye out for security, performance, and stability issues in Sails, Waterline, and Skipper and dealing with them as they arise, but we're not actively developing new features until the other necessary infrastructural work is complete. We believe this is the most important thing for the health of the framework long-term.

Node-Machine Project

• If you aren't familiar with machines, basically they're the next generation of blueprints.
• Aside from the related generators, adapters, and hooks in Sails, we're maintaining the machine runner, mp (command-line tool), a test runner, as well as a set of core machinepacks. Fortunately, these projects have tens of thousands of tests, which has helped them stay very well-behaved.
• rttc (type system and validator) -- this is like anchor, but gets down to a lower level and makes strong, conventional type guarantees.
• machine-as-script lets you run any machine as a command-line script
• Most recently, we've pushed some updates to machine-as-action. This module lets you run any machine as a Sails/Express action. The incoming request is parsed to provide the machine's arguments, and the response is encoded as JSON by default (just like blueprints). And as of this week, you can now also consume file uploads (i.e. upstreams) and respond with HTML views, redirects, or stream raw data (e.g. for file downloads).

Treeline

• The next release of Treeline overcomes some of the major roadblocks present in the current version, plus some of the stickiest limitations of blueprint-only development in general:
  • branching (think of config/policies.js, but as a tree)
  • generics (variables without strict type schema guarantees)
  • convergence (think of async.if() or try..catch..finally)
  • subcircuits (think of the function you pass in to _.each())
• We've also been focusing on the user experience. We ripped out the dependency on Angular and did a wholesale redesign of the interactive JSON/expression editor to solve bugs and timing issues w/ the digest loop, and we rearchitected the main workspace UI with an emphasis on all-keyboard navigation (drag and drop is cool, but once you know what you're doing, you need to be able to work as quickly as you can in Sublime)

Contributing

As far as contributing, the single-biggest thing we could use some help with right now is tests: specifically complete integration tests for different configurations (e.g. create sails app in a folder with config/views.js set to layout: false, lift the app, send a request to a route which serves a view, etc).

Alternatively, if you're not comfortable with tests, the best place to chip in is in our reference documentation. Currently, about 50% of the pages have been updated to use the new format (e.g. blueprint actions, config props, methods), but many (e.g. blueprint actions,methods) still have not.

Official Announcements

We use the @sailsjs Twitter account to link to official announcements; but for the long-form content, I've always just used a gist or my personal blog. Historically, a lot of the communication from our core team has been decentralized and organic; mainly because that's the way we tend to operate (which is good sometimes, and not so good others). If there's interest, I'm happy to move that communication to an official blog.

@codymorrison hope that helps clear things up, and presents some places where we could use some help. Thanks for your kind words and support.
-Mike

@mikermcneil

That is a lot of good information and I want to thank you for taking the time to address these concerns, but I think out of the 3 points @codymorrison listed, only the last one was casually addressed. These first two were sort of left out:

• Lack of communication with contributors and their pull requests (including things like an Express 4 upgrade and deep populating of associations
• An apparent loss of active contributors which are now creating a rival project, and with it several valuable 3rd party components.

@mikermcneil I would have to agree with @codymorrison and @randallmeeker, these seem to be some pretty big concerns for anyone developing a sails applications right now.

I'd also mention the fact that @sailsbot has created some controversy within the community (#3423) essentially closing nearly every issue regardless of status with pretty much all balderdashy/sails projects, while the community has created new issues that get closed by @sailsbot or with core maintainers like @devinivy reopening them more than once, but not getting any feedback.

This has all kind of taken place behind the scenes a bit and only those who really are following these projects are aware of issues such as this one, but it would be nice to understand what is really happening here and the current vision for sails since it seems a bit unclear other than it has been put on hold, which is likely why some core developers have moved on to working on trailsjs (an ES6 rewrite of sails)

@animedbz16 @codymorrison

which is likely why some core developers have moved on to working on trailsjs (an ES6 rewrite of sails)

An apparent loss of active contributors which are now creating a rival project, and with it several valuable 3rd party components.

Guilty :( Trust me, we did not take this decision lightly. My company and I have been Sails.js users and contributors since mid-2014, and have a lot of knowledge and code and effort invested in this framework. We maintain two of the more popular Sails addons (sails-auth and sails-permissions), and it's painful to watch the atrophy of the framework on which that work is based.

As a "former" contributor and maintainer of Sails.js who tried to keep the community on life support until Mike McNeil kicked me out, my company has indeed abandoned usage and development on the Sails.js platform. Mike is in fact really busy, but this is the problem, not some side of effect. It does create an issue of transparency and communication. Perhaps you've also noticed that literally the only issues he responds to are issues asking whether Sails is dead. He rappels down from the ivory tower, proclaims "Sails is not dying, never dear", and then disappears. This is not transparency, and it's not commitment. It's lame.

and it's part of why we've been in more of maintenance/stability mode with Sails and Waterline over the past 9 months.
To set the record straight, here's a quick look at what we're currently working on

Of all the things they're working on, Sails is most definitely not one of those things. We don't need someone to "set the record straight" when we can just look at the actual commit logs. There are a measly 83 commits in the past 9 months. Most of those are documentation/README changes, and many of the rest were from myself or other members of my team who are now working on trails.js. I don't know about you, but in my world, 9 months is a long time. They've made a couple token 0.11.x patch releases recently to make it look like the framework is actively maintained. It is not.

This community really is full of awesome people, but ultimately there's one single person who holds the keys to everything. Someone who apparently thinks they will be less busy tomorrow than they are right now, and someday will be able to actually maintain Sails.js. We can't entrust our entire software company to a single individual. This is why we're moving on.

We're not going far, however. We're working on a new framework with an upgrade path from Sails.js: https://github.com/trailsjs. We're building the framework around modern tools (hapi, webpack) and a more modular, focused core, written in ES6 from the ground up. If you're interested in joining our community, we'd love for you to join us. There will be some awkwardness in the transition, but we're prepared to see it through. More important than the improved technology and design is the design and structure of the community. We're happy to share responsibility and control with contributors. We have committed several full-time developers to the project who are devoted to designing and building Trails.js. Looking forward to working with you all :)

❤️

I think this is very disconcerting. I understand that everybody has to feed themselves but their is a community that has developed around this platform. Whether you are a service based consultancy or a product company you are the sponsoring organisation of the framework and hold the keys.

It would be great to see a bit more activity on Sails

It's very hard for the non-contributors to criticise (shouldn't we just be grateful?), but we have two choices: (i) give our feedback anyway or (ii) switch to another framework (i.e. neither sails or trails).

The situation we're observing requires at least two parties with their side of the story. We note that @mikermcneil isn't prone to public discussion in order to avoid exacerbating the problem as well as giving a negative public image (of himself, his company and/or the services and products being built).

On the other side, @tjwebb seems to promote transparency, voicing his side of the story... and positioning trails as the new sails.

Both "strategies" aren't optimal: you're both giving negative signals to any responsible developer who decided to bet on sails.

Side notes: you're not alone. Has anyone noted a similarity with what happened with node.js 0.x and iojs? Now they're back together, because it's orders of magnitude easier. If they could overcome their issues, I'm sure you can too. Open source is great, but unmaintained projects tend to die at light speed, especially when a drop in replacement arrives and promises the Truth (am I referring to trails?). Angular's UI Bootstrap vs AngularStrap... felt exactly like the current situation! As you can see both projects are still alive, doing a great job duplicating each other's functionality (without copying code, but who cares?). Look for EventEmitter, EventEmitter2 and EventEmitter3... examples are countless and you probably can think of better ones than me.

Why not keep all the good of sails and bring all the good trails is promising? I don't think sails is old, it's just not maintained enough! With a better waterline (please merge deep populate), the potential is huge (please also merge the support for Express 4).

When I read about your internal issues, the first reaction was to look for another framework (as I use sails only for the API, and it does a great job for prototyping but it's not impossible to replace)... soon to realise all the good reasons I picked it in the first place, which started with its community and modularity.

There's no doubt @tjwebb and your core devs on trails can make quick progress as you can reuse lots of sails knowledge (even if it's not an explicit fork). The current code feels quite sails-y (from a usage perspective, the models and project structure look similar, and so on)... we'll need to see the code in January to see the real difference. But trails won't be "battle-tested" before a long time (years).

I think that if you do not solve your issues, two rival frameworks will compete for the same developers, which is valid in itself, but it doesn't mean you're not wasting resources (time, energy, money, ...).

I wish you could just sit around a table and find a solution together, otherwise it means that we should go see if we'd be _hapi_er... (sorry for the pun 🐙).

All of that being said, thank you for all the sweat, the great work and sharing in the first place. I should probably go back coding now, I talk too much.

Hey hey hey everyone. I can chime in on Waterline since everyone keeps mentioning deep populate! The PR for the polyfill is pretty much ready to go, it's a recursive query runner that will run queries until the results are completed. This will increase the query count in sql adapters until the adapters understand how to interpret and build these join queries (nosql adapters don't have joins so those will work). I tested it a while back and it looked almost all the way there but needed a bit more work. I can try and get that merged in master by the end of the year, could use more eyes on it!

Getting the adapters to understand how to run that query is VERY important though. Once released anyone using a sql adapter will see decreased performance when using deep populate vs populate (should be better than what you have to do now though) and I've been spending my free time working on a replacement for our sql query builder (waterline-sequel) that makes generating those queries easy.

Part of this work involves a new declarative intermediate data structure that is used to talk between Waterline and the adapters. Right now we only have the criteria but really need a way to express create, update, delete queries etc in a normalized way. Hopefully we can get something that we can use Knex for the actual sql query building and deprecate waterline-sequel.

I'm more involved in Waterline but as far as Sails goes it's still maintained. Sure commits may have slowed down but any critical bugs have been fixed in a timely manner. Express 4 is a big open PR but I don't think it will provide any additional functionality or security patches to the framework. Most of the features people request are Waterline features, most features in Sails can be implemented as hooks. Sorry if I'm not as up to date on any outstanding issues there.

I wish you could just sit around a table and find a solution together, otherwise it means that we should go see if we'd be hapier... (sorry for the pun 🐙).

I really wish that's what could've happened. Unfortunately we (me and my team) tried and failed for nearly a year to convince McNeil that we're actually all on the same team here and share the project. He seemed more interested in retaining control -- indeed, when we pushed the issue again recently, he decided to kick us all out of the "balderdashy" github organization. So we were at the table, and @mikermcneil called security to escort us out of the building. McNeil has still never talked to me directly about why he kicked us out, but that decision seems to have come from a place of self-interest, not with the interests of the community in mind.

One person has decided to position himself as the Monarch of Sails, and we no longer have faith that that person has the best interest of the community in mind. Indeed, he admits above that he's running a "product company" and is no longer in the software services business. It is for these reasons that our team has decided to create trails.js.

I don't think sails is old, it's just not maintained enough!

That's partially true. Active maintenance would help, but after working on (and in) the project for over a year, we've concluded that there are serious technical deficits that would take a lot of work to fix. Also, because Sails is overly monolithic, many of these issues simply can't be solved without significant breakage. We decided to create a new framework based on modern tools and lessons learned from working with Sails.

@particlebanana

Hopefully we can get something that we can use Knex for the actual sql query building and deprecate waterline-sequel.

All of the adapters in https://github.com/waterlinejs are using Knex. If you'd like help with this effort, the Trails.js team would be happy to assist. I'm also a maintainer of knex, so this is an effort we're very interested in continuing.

Express 4 is a big open PR but I don't think it will provide any additional functionality or security patches to the framework.

Seeing the maintainers make statements that are provably inconsistent with reality doesn't really inspire confidence. From http://expressjs.com/en/3x/api.html:

I'm more involved in Waterline but as far as Sails goes it's still maintained.

Maintained. You all keep using that word. I do not think it means what you think it means.

This is very concerning. @tjwebb has always responded to my mails in a matter of minutes. :/

Ok I'm not here to pick a fight, so I'm locking this before it gets out of hand. This isn't the place for personal attacks, it's just a framework!

@tjwebb the adapters you have are still using wl-sql for the query generation it looks like. I've been working on a new criteria format that can be sent into Knex fairly easily. I still need to make sure it's going to work for nosql databases such as mongo but then I'll post it for feedback from everyone and would love help getting that transitioned over.

re: Express 4 and deep populate

That is a lot of good information and I want to thank you for taking the time to address these concerns, but I think out of the 3 points @codymorrison listed, only the last one was casually addressed. These first two were sort of left out

@randallmeeker Please refer to the discussion in those pull requests. re: Express 4, see #3235 (comment).

re: sailsbot

@sailsbot has created some controversy within the community (#3423) essentially closing nearly every issue regardless of status with pretty much all balderdashy/sails projects.

@animedbz16 The reality of open source is that the vast majority of issues created in our repos are questions or feature requests. Sailsbot closes issues which have had no activity for over 30 days. It also ignores confirmed bugs, and very clearly states when closing issues that the issues should be reopened if there is a real bug that has gone unaddressed. In addition, closed issues still show up in search results, and comments can still be posted. This approach has been very effective so far; it allows us to keep an eye on all new issues that are posted, and to be able to tell more quickly if major issues are posted.

Edit: Sailsbot herself jumped in to share her thoughts here.

re: Trails

@devnomad @tommykennedy @coderdecoder :
I know this issue is being linked to a lot, and that there's been a lot of confusing information posted across the ether the past couple of weeks. I've learned a lot from this experience, some of which I'll share on my personal blog in coming weeks.

However, there are a few statements above I cannot leave unaddressed.

An apparent loss of active contributors which are now creating a rival project, and with it several valuable 3rd party components.

Sails and Waterline are maintained by me and the rest of the core team, just as they have been for the last four years since I wrote them. The only contributor that was removed was Travis Webb- someone who was only added in mid-2015. I don't want to discount the work that Travis did-- I used to consider him my friend. But it's important to understand that Travis was not materially involved in the development of the Sails framework, and that he is the only former Sails contributor (i.e. someone with write access to our repos) working on Trails.

some core developers have moved on to working on trailsjs (an ES6 rewrite of sails)

Again, Travis Webb was also the only contributor removed from Sails, in late November. Ever since, he has been doing everything he can to discredit our project, including claiming that the Sails team is working on Trails, or that Sails is "abandoned" or "dead". I'm sorry that this played out in public; and I'm sorry our issues were used as a pulpit-- it won't happen again. The whole situation is unprofessional and childish, which is part of the reason I've been mostly silent up until now.

All of the adapters in https://github.com/waterlinejs

This is a GitHub organization created by Travis that is improperly using the Waterline logo and has no official affiliation with the project. Same thing with the "waterlinejs" Twitter account.

They've made a couple token 0.11.x patch releases recently to make it look like the framework is actively maintained.

I feel very hurt by this. We've released patches for bugs and to release important updates, such as Node v4 support in 0.11.2. As I said before, our focus for Sails core is security, stability, and performance.

I think there is a growing sentiment around Sails.js that its dying and being abandoned by the core team and I keep feeling its going to cause a lot of harm to the ecosystem that surrounds Sails in general.

Sails was downloaded more times in November than any other month in its history. As a point of reference, I'm the CEO and founder of a Y Combinator company that runs our app in production on Sails, as is another company from our batch (completely by chance). We are heavily invested in Sails, as are many other companies and developers all over the world.

I have no problem with forks-- they're a great way to make customizations that you need without convincing our core team to merge them into the Sails code base. But I think it's important to realize that the Trails rewrite is not about technical issues; it's a personal vendetta against me and our project.

To bring some closure to this issue: the rest of the core maintainers and I are as committed as ever to the continued development and support of Sails and Waterline. If you have questions or concerns about our team, what we're working on, or where we're headed, please don't hesitate to reach out to me directly.

EDIT: Updated links.