Hey there Gulp lovers!
I would like to share with you my first hands-on experience with Gulp. Hopefully, it’ll be helpful for those who are also new to this handy tool that can be quite frustrating sometimes. Before I go ahead and present my Gulp solution I should first present the problem and the need for it. So, here we go.
When Crom stops loving you
My team mates and I have been using a nifty Chrome extension that helped us numerous times when it comes to merging and minifying files. Unfortunately, as of last week, this extension stopped working after an update to Chrome, version number 43 and odd. We quickly discovered that whoever had an older version of Chrome had the extension working just fine. We must have clearly missed some big announcement about policy or rule changes to the way Chrome handles these things. We simply didn’t have time to investigate this. After all, it could have been a bug on their end or simply something we’ve had to fix in our extension so it would be compatible with the new version since this was an in-house built extension. For those who are wondering about this extension, here is a link to it.
So, now we have this situation where having felt quite comfortable and victorious for the last 2 years, we have found ourselves at the mercy of multiple unminified files and begging for help from Crom. One can not depict our need better than this :
Options, Kowalsky!
I’m a big fan of Penguins of the Madagascar and I quickly remembered what Skipper often asks Kowalsky. Options! What easily comes to mind is to fix the extension but we didn’t know where to start because the guy who built it left the company long time ago. We could have taken a stab at it but that would mean to investigate a great deal of how things work in or with Chrome. We soon realized that this option was costly.
What came next wasn’t particularly helpful either. We kept looking for off-the-shelf products and nothing seemed to be a good candidate since we rely on a very specific folder structure either in local or server file system. We have a bunch of files that are stored in a central spot so only a few necessary project files could be added to PHP files to extend the core functionality. It was clear at this point that we had to develop a way to fix our problem. The fact that core files have been distributed in many folders also depicts the very essence of the problem which brings us to…
Lemme see, lemme see
function VORTEX($separate, $minified){
if ($GLOBALS['ENVIRONMENT'] == "LOCAL")
echo $separate;
else
echo $minified;
}
The piece of code shown above is what lets us use unminified separate files when we are displaying our pages in a local environment. When the page goes to server it displays the minified part. Trivial, and here is how we use it before we close the “head” tag.
<?php
VORTEX('
<link rel="stylesheet" type="text/css" href="'.URL_STATIC_LOCATION.'reset-min.css">
<link rel="stylesheet" type="text/css" href="'.URL_STATIC_LOCATION.'style.css">
','
<link rel="stylesheet" type="text/css" href="'.URL_STATIC_LOCATION.'style.min.css" />
');
?>
We usually place another VORTEX call near the bottom before we close the “body” tag for Javascript references. Here it is.
<?php
VORTEX('
<script type="text/javascript" src="'.LOCALHOST.'/Resources/jquery.cookie.js"></script>
<script type="text/javascript" src="'.LOCALHOST.'/Resources/jquery.swfobject.1-1-1.min.js"></script>
<script type="text/javascript" src="'.LOCALHOST.'/Resources/TIN Framework/tin.core.2.0.js"></script>
<script type="text/javascript" src="'.LOCALHOST.'/Resources/TIN Framework/tin.device.1.0.js"></script>
<script type="text/javascript" src="'.LOCALHOST.'/Resources/TIN Framework/tin.utils.2.0.js"></script>
<script type="text/javascript" src="'.LOCALHOST.'/Resources/TIN Framework/tin.apis.1.0.js"></script>
<script type="text/javascript" src="'.LOCALHOST.'/Resources/TIN Framework/tin.datatype.csv.1.0.js"></script>
<script type="text/javascript" src="'.LOCALHOST.'/Resources/TIN Framework/tin.data.2.0.js"></script>
<script type="text/javascript" src="'.LOCALHOST.'/Resources/TIN Framework/tin.validators.2.0.js"></script>
<script type="text/javascript" src="'.LOCALHOST.'/Resources/TIN Framework/tin.device.1.0.js"></script>
<script type="text/javascript" src="'.LOCALHOST.'/Resources/jQuery Components/jquery.touchSwipe.min.js"></script>
<script type="text/javascript" src="'.LOCALHOST.'/Resources/jQuery Components/jquery.mousewheel.js"></script>
<script type="text/javascript" src="'.LOCALHOST.'/Resources/jQuery Components/numericstepper/jquery.numericstepper.1.0.js"></script>
<script type="text/javascript" src="'.LOCALHOST.'/Resources/jQuery Components/jquery-ui-1.10.3.widget.min.js"></script>
<script type="text/javascript" src="'.LOCALHOST.'/Resources/jQuery Components/locationpicker/locationPicker.js"></script>
<script type="text/javascript" src="'.URL_STATIC_LOCATION.'script.js" id="'.PL_NAME.'"></script>
','
<script type="text/javascript" src="'.URL_STATIC_LOCATION.'script.min.js" id="'.PL_NAME.'"></script>
');
?>
Let me explain in more detail what those LOCALHOST and URL_STATIC_LOCATION blocks mean. We use Evolphin Zoom for version management. When we pull the files they go under /Users/username/Documents/Evolphin Zoom/. Our team has been using common resources put under Resources folder. Looking at the file reference makes me think it might need a better organization but that’s a different fight. The beauty of VORTEX system is that you may organize your local folders the way you want and as long as they are all minified in one file it would not matter. So, LOCALHOST is simply a reference to your local Zoom folder so Resources can be reached.
URL_STATIC_LOCATION is a different story on the other hand. Its value changes depending whether you are running the code in your local or on the production server. In a production environment its value is calculated based on the project configuration. Luckily, our team doesn’t take care of that and we don’t need to know its value but we simply have to add as a prefix. Otherwise, all projects would use one and only one script.js file. We don’t want that. However, we don’t want to calculate that value for each folder we check out from Zoom either so URL_STATIC_LOCATION value is set to an empty string when it’s running in local. Therefore, each folder can use its own script.js file. Similarly, the VORTEX block that serves CSS styles is using relative style files.
We have to eventually figure out what those variable resolve to so our Gulp tool can do its job. Most Gulp tutorials I have found only talks about combining and minifying relative files and it runs in a folder where you have your “soon to be minified files”. I wanted to approach this differently though. If possible, I wanted to run our Gulp tool in /Resources/Gulpible/ as a watch event. It would make sense at this point to reveal some of our projects’ folder structure to shed light into how Gulp will do its job.
- /Users/kumsalo/Documents/Evolphin Zoom/Domain Name 1/com/Tours/tour x/work/
- /Users/kumsalo/Documents/Evolphin Zoom/Domain Name 1/com/Tours/tour y/
- /Users/kumsalo/Documents/Evolphin Zoom/Domain Name 2/com/Tours/tour x/work/
- /Users/kumsalo/Documents/Evolphin Zoom/Domain Name 3/com/Tours/tour x
- /Users/kumsalo/Documents/Evolphin Zoom/Domain Name 3/com/Tours/tour z
As you can see, Zoom root folder is partitioned into different domain names and each is followed by the domain extension. Then comes the project type as we call them Tours. Due to human ignorance the folder name “Tours” can sometimes be Tour, tour or tours. We’ll have to account for that when we add files to gulp watch. Our team’s guidelines clearly dictates that we should use a work and design folder under a project folder so under normal conditions we should have a work folder for each one of those tour x,y and z folders but human ignorance comes into play again. So, we don’t always have that folder as well. Let’s add that to our list of carefully watched files.
I want to see the real deal
var gulp = require('gulp');
var concat = require('gulp-concat');
var jsmin = require('gulp-jsmin');
var uglify = require('gulp-uglify');
var sourcemaps = require('gulp-sourcemaps');
var fs = require('fs');
var watch = require('gulp-watch');
var minifyCSS = require('gulp-minify-css');
var Download = require('download');
var del = require('del');
var LOCALHOST = "";
var pathToWork = "";
var files = {};
var extractRemotePaths = function (element, index, array) {
if (element.search("http://") > -1 || element.search("https://") > -1) {
//we have to add this remote element to "to be downloaded" list
files.remotePaths.push(element);
var remoteFileName = element.substr(element.lastIndexOf('/') + 1, element.length);
//we have to update the reference of this remote file in our paths list because after the download the file will be available locally
array[index] = './remoteFiles/' + remoteFileName;
}
};
Let’s get the variable declarations out of the way first. There is nothing extraordinary here except extractRemotePaths. That function is supposed to recognize external paths and add them to the download list. After the download list gets pushed a new element then the same element that exists in the original file array is updated on line 25 so gulp can access it from the file system. Basically, gulp doesn’t download the external path. Instead, we have to first prepare a download list, download each dependency and finally tell Gulp the new file name and reference so it can access it. It’ll make more sense once we see more meat.
gulp.task('minify', function () {
console.log("Starting Gulping");
['cssify','scriptify'].forEach(function (element, index, array) {
console.log("We'll " + element + " the following files:", files[element].paths);
var stream = gulp.src(files[element].paths)
.pipe(concat(files[element].minifiedFileName))
.pipe(element === "cssify" ? minifyCSS() : uglify())
.pipe(gulp.dest(pathToWork));
stream.on('error', function(err){
console.log("WTF Error:", err);
});
stream.on('end', function(){
console.log(element.toUpperCase(), " DONE!");
});
console.log("Minified file:", files[element].minifiedFileName);
});
});
This is where things start to get interesting. Most of you who have worked with Gulp to combine files will recognize lines 35-38. We have a twist here. Since almost every PHP file will have 2 VORTEX blocks, one for CSS and one for Javascript, we are going to run the gulp stream on an array of 2 elements. Paths that go into the stream are separated in “cssify” and “scriptify” blocks in files object. Each one of these objects will have their own path so if we take the first VORTEX block we defined previously we should have 2 file references in files.cssify and a handful of file references in files.scriptify. Also, line 36 shows that each files object will carry its own minifiedFileName. This is basically the name that comes from the second block of VORTEX, basically that’s what echo $minified; does. We’ll look into how to get that value soon. So, essentially, link blocks will be minified into style.min.css since they are stored in files.cssify and script blocks will be minified into script.min.js…
A basic ternary check on line 37 will do the trick to pick the right minification tool depending on the element type. Here, you can pick jsmin over uglify. I deliberately left both references in the variable declaration section in the top part of the code. Finally, gulp will need a destination to write the result file. Defining the value of pathToWork and minifiedFileName will be the juiciest part of this whole process but it’s nothing more than a few regular expressions look up. So, shall we?
Gulp is watching you
Before we get to analyze how the watch task monitors changed files and extracts paths I would like to quickly look at how Gulp will handle remote files since that’s sort of related to gathering files to minify.
gulp.task('gatherRemoteFiles', function () {
var numberOfDownloadedRemoteFiles = 0;
files.remotePaths = [];
files.cssify.paths.forEach(extractRemotePaths);
files.scriptify.paths.forEach(extractRemotePaths);
if (files.remotePaths.length) {
files.remotePaths.forEach(function (element, index, array) {
new Download({
mode: '755'
})
.get(element, './remoteFiles/')
.run(function () {
numberOfDownloadedRemoteFiles++;
if (numberOfDownloadedRemoteFiles == array.length)
gulp.start('minify');
});
});
} else {
gulp.start('minify');
}
});
To download the remote files, gatherRemoteFiles uses extractRemotePaths. The watch task will fill the files object which may have remotePaths and extractRemotePaths will take care of extracting each path accordingly for CSS and Script files. Once the remote paths are collected all we have to do is count the number of files we have downloaded and then trigger the minify task which is essentially the same merge/minify example you can find in most tutorials. We’ve already seen how we can switch between two types, link and script. Essentially, all Gulp is doing in gatherRemoteFiles is to create local references of the remote paths so it can do its job. Finally, we can get to see the file reference collection in the watch task:
gulp.task('watch', function () {
var watchedFiles = [];
['../../**/Tours/**/', '../../**/Tour/**/', '../../**/tours/**/', '../../**/tour/**/'].forEach(function (element, index, array) {
watchedFiles.push(element + '*[^.min].js');
watchedFiles.push(element + '*[^.min].css');
watchedFiles.push(element + 'index.php');
});
watch(watchedFiles, function (vinyl) {
pathToWork = vinyl.path.substr(0, vinyl.path.lastIndexOf('/') + 1);
LOCALHOST = pathToWork.substr(0, pathToWork.search('Evolphin Zoom') + ('Evolphin Zoom').length);
fs.readFile(pathToWork + 'index.php', "utf-8", function (err, data) {
if (err) throw err;
var VORTEX_BLOCKS = data.match(/VORTEX\('[\s\S]*?'\);/g);
/*
we have to clean each block since matched data blocks have escaped characters
then we'll convert each block to an array of 2 elements in which the first block
will hold different chunks to minify whereas the second block will be the minified file name
*/
VORTEX_BLOCKS.forEach(function (element, index, array) {
var blockType = element.search("<link") > -1 ? "cssify" : "scriptify";
files[blockType] = {};
//sanitize the urls
element = element.replace(/'.LOCALHOST.'/g, LOCALHOST);
element = element.replace(/'.URL_STATIC_LOCATION.'/g, pathToWork);
//sanitize the escaped characters
element = element.replace(/\n\t/g, "");
//there will be two main blocks, one for separate and one for minified file declaration
var blocks = element.split("','");
//we have to split those separate file references either for links or scripts
files[blockType].paths = blocks[0].match(/(?:href=|src=)["|']([^"]*)["|']/g);
files[blockType].paths.forEach(function (currentValue, index, array) {
array[index] = currentValue.replace(/href=/, "").replace(/src=/, "").replace(/"/g, "");
});
//finally we can decide which file name will be used after we combine and minify separate files
var minifiedFilePath = blocks[1].match(/(?:href=|src=)["|']([^"]*)["|']/g)[0].replace(/href=/, "").replace(/src=/, "").replace(/"/g, "");
files[blockType].minifiedFileName = minifiedFilePath.substr(minifiedFilePath.lastIndexOf('/') + 1, minifiedFilePath.length);
});
//start off with cleaning up the existing remoteFiles directory
del('./remoteFiles/*', {
force: true
}, function (err, paths) {
console.log('Deleted remoteFiles in Gulpible');
gulp.start('gatherRemoteFiles');
});
});
});
});
Now, this is a lot of code for a simple watch task and that’s because we have a complex path structure with all those dependencies. Let’s break it down a bit.
Line 84 listens to 4 different folder structures but they are essentially the same, remember human negligence? watchedFiles array will be pushed file references for all script files (except the ones that end with .min.js), all style files (except the ones that end with .min.css) and index.php. So, since all style and script files under a work folder might necessitate a reminification we add them to the watch list. Also, any change to the index.php should trigger a reminification because we might change the order of script blocks.
As I mentioned before, this task is responsible for finding out which work folder triggers a need for minification. The callback function for the watch list starts with defining the value of pathToWork so gulp streams can do their job in minify task as shown above (line 35). We are also updating the value of LOCALHOST based on the value of pathToWork. We could have hardcoded this value initially but since the team is going to use the same minification process, I wanted to introduce a way so LOCALHOST can take its value from the user’s path to the work folder. It’s basically “/Users/kumsalo/Documents/Evolphin Zoom/” with the username picked for you by the watch process.
The rest of the watch operation is one long block of callback for node file system. Once we figured out “pathToWork” all we have to do is to read the VORTEX blocks in index.php and separate them into css and script blocks in “files” object (line 15). At line 98 we collect VORTEX blocks with a simple regular expression. At this point we ended up fixing some of our VORTEX blocks so they respect the necessary end of line and tab characters.
Line 105 will run over two blocks of VORTEX. The first line in that forEach block defines the type of block which be used later in “files” object. Sanitization takes place here to replace PHP variables in our VORTEX blocks. I believe the code I provided above has useful comments so I’ll briefly summarize it by saying that at the end of watch task we are supposed to have a “files” object filled with “cssify” and “scriptify” objects; each holding necessary path information for gulp to put in the stream. Since there might have been previously run tasks and downloaded files from previous sessions we trigger a clean up which starts the cycle :
watch -> gatherRemoteFiles (extractRemotePaths) -> minify -> DONE
Final Words
This whole thing is actually a different version of normal Gulp process of watching files, adding them to the stream and exporting minified files. In our case, there are a handful of framework files that are kept separately. So, all this fussing with paths and downloading remote files are just extra but necessary steps. I hope my explanation was as clear as daylight and it will help someone out there with similar needs.
It’s customary to share the whole code so I’ll do that here by following it with a screenshot that shows what happens when it’s running.
var gulp = require('gulp');
var concat = require('gulp-concat');
var jsmin = require('gulp-jsmin');
var uglify = require('gulp-uglify');
var sourcemaps = require('gulp-sourcemaps');
var fs = require('fs');
var watch = require('gulp-watch');
var minifyCSS = require('gulp-minify-css');
var Download = require('download');
var del = require('del');
var LOCALHOST = "";
var pathToWork = "";
var files = {};
var extractRemotePaths = function (element, index, array) {
if (element.search("http://") > -1 || element.search("https://") > -1) {
//we have to add this remote element to "to be downloaded" list
files.remotePaths.push(element);
var remoteFileName = element.substr(element.lastIndexOf('/') + 1, element.length);
//we have to update the reference of this remote file in our paths list because after the download the file will be available locally
array[index] = './remoteFiles/' + remoteFileName;
}
};
gulp.task('minify', function () {
console.log("Starting Gulping");
['cssify','scriptify'].forEach(function (element, index, array) {
console.log("We'll " + element + " the following files:", files[element].paths);
var stream = gulp.src(files[element].paths)
.pipe(concat(files[element].minifiedFileName))
.pipe(element === "cssify" ? minifyCSS() : uglify())
.pipe(gulp.dest(pathToWork));
stream.on('error', function(err){
console.log("WTF Error:", err);
});
stream.on('end', function(){
console.log(element.toUpperCase(), " DONE!");
});
console.log("Minified file:", files[element].minifiedFileName);
});
});
gulp.task('gatherRemoteFiles', function () {
var numberOfDownloadedRemoteFiles = 0;
files.remotePaths = [];
files.cssify.paths.forEach(extractRemotePaths);
files.scriptify.paths.forEach(extractRemotePaths);
if (files.remotePaths.length) {
files.remotePaths.forEach(function (element, index, array) {
new Download({
mode: '755'
})
.get(element, './remoteFiles/')
.run(function () {
numberOfDownloadedRemoteFiles++;
if (numberOfDownloadedRemoteFiles == array.length)
gulp.start('minify');
});
});
} else {
gulp.start('minify');
}
});
gulp.task('watch', function () {
var watchedFiles = [];
['../../**/Tours/**/', '../../**/Tour/**/', '../../**/tours/**/', '../../**/tour/**/'].forEach(function (element, index, array) {
watchedFiles.push(element + '*[^.min].js');
watchedFiles.push(element + '*[^.min].css');
watchedFiles.push(element + 'index.php');
});
watch(watchedFiles, function (vinyl) {
pathToWork = vinyl.path.substr(0, vinyl.path.lastIndexOf('/') + 1);
LOCALHOST = pathToWork.substr(0, pathToWork.search('Evolphin Zoom') + ('Evolphin Zoom').length);
fs.readFile(pathToWork + 'index.php', "utf-8", function (err, data) {
if (err) throw err;
var VORTEX_BLOCKS = data.match(/VORTEX\('[\s\S]*?'\);/g);
/*
we have to clean each block since matched data blocks have escaped characters
then we'll convert each block to an array of 2 elements in which the first block
will hold different chunks to minify whereas the second block will be the minified file name
*/
VORTEX_BLOCKS.forEach(function (element, index, array) {
var blockType = element.search("<link") > -1 ? "cssify" : "scriptify";
files[blockType] = {};
//sanitize the urls
element = element.replace(/'.LOCALHOST.'/g, LOCALHOST);
element = element.replace(/'.URL_STATIC_LOCATION.'/g, pathToWork);
//sanitize the escaped characters
element = element.replace(/\n\t/g, "");
//there will be two main blocks, one for separate and one for minified file declaration
var blocks = element.split("','");
//we have to split those separate file references either for links or scripts
files[blockType].paths = blocks[0].match(/(?:href=|src=)["|']([^"]*)["|']/g);
files[blockType].paths.forEach(function (currentValue, index, array) {
array[index] = currentValue.replace(/href=/, "").replace(/src=/, "").replace(/"/g, "");
});
//finally we can decide which file name will be used after we combine and minify separate files
var minifiedFilePath = blocks[1].match(/(?:href=|src=)["|']([^"]*)["|']/g)[0].replace(/href=/, "").replace(/src=/, "").replace(/"/g, "");
files[blockType].minifiedFileName = minifiedFilePath.substr(minifiedFilePath.lastIndexOf('/') + 1, minifiedFilePath.length);
});
//start off with cleaning up the existing remoteFiles directory
del('./remoteFiles/*', {
force: true
}, function (err, paths) {
console.log('Deleted remoteFiles in Gulpible');
gulp.start('gatherRemoteFiles');
});
});
});
});
// The default task (called when you run `gulp` from cli)
gulp.task('default', ['watch']);
