Musings and Solutions for SharePoint, O365, and Azure

2007 2010 2013 2016 ALM Development Javascript Language SharePoint Online SharePoint Version TypeScript

Conquer your dev toolchain in ‘Classic’ SharePoint – Part 2

January 9, 2018

Conquer your dev toolchain in ‘Classic’ SharePoint – Part 2

This entry is part 2 of 4 in the series Conquer your dev toolchain in 'Classic' SharePoint

In the first post in this series I discussed some of the benefits of formalizing your client-side development process and then a bit about starting the process of tooling up. A common scenario to develop our own client-side solutions in SharePoint is to point a Content Editor (CEWP) or Script Editor (SEWP) web part at our custom html, css, and js files that are sitting in a document library somewhere in our environment. In this post I want to dive into the most basic implementations of a development toolchain that will automatically deploy those files into a location in SharePoint. This does not mean, and in most cases, should not mean, “production”. What it means is that while you’re developing your code any changes you make will be automatically uploaded to a location that you already have your CEWP or SEWP pointed to, that way when you refresh the page your custom solution will refresh with the latest version of your code. To accomplish this, we’re going to use Gulp which is a JavaScript based task runner similar to the build process that exists in Visual Studio.

As I said, I’m going to be sharing with you now the most basic package.json and gulpfile.js to configure this process. In addition, I’ve created 2 settings files, one has the project settings and the other has security and local configuration values that my gulp file will use. The reason I create two files is that one, settings.json, I check into our local source control and the other, settings_security.json, I do not. This way in order for me to get up and running with a project that someone else worked on all I have to do is create a local copy of settings_security.json and run “npm i” which will install all the dependencies I added to my package.json file.

Using REST to move your files into SharePoint – SharePoint 2013 and beyond

First let’s look at our package.json file for deploying files to SharePoint 2013 -> Office 365. This package.json and it’s corresponding gulpfile.js work with any SharePoint environment that supports the REST apis.

The only part of this file we’re going to discuss is the devDependencies section. These are all the packages we’ll need to support our deployment. I’ve linked to them and provided a very brief description.

gulp The task runner
gulp-cache Caches files so that we can tell if the file has changed and only upload if it has.
gulp-watch Watches a particular set of files paths for changes.
map-stream Provides the ability to pass each file that changed from the watch into spsave to be uploaded to SharePoint
spsave Uploads the file into a SharePoint library.

One other thing I’d like to point out in my devDependencies is the use of the ~ before the version number. This is a modification I made purposefully, again due to run ins with older code getting updated and not working together very nicely. The ~ tells npm that it should “Allows patch-level changes if a minor version is specified on the comparator. Allows minor-level changes if not.” By default, when you use the ‘npm i’ to install a new package it will use the ^ range which “Allows changes that do not modify the left-most non-zero digit in the [major, minor, patch] tuple.” What I’m basically saying is that I would go back to code I had written a year ago, pull it out of source control and do an ‘npm i’. Because most of the packages I had reference updated since then I would get the latest version per the ^ range rules. What I found is I would often end up with breaking changes and it was super frustrating to go back and fix them for something that wasn’t broken to begin with, so I made this modification to all my package.json files.

Ok, so if you’ve run “npm i” then all of these dependencies (and their dependencies) should be installed in the node_modules subfolder. Once that is complete we can use them by creating a gulpfile.js which is basically a JavaScript file that uses these packages and runs on our NodeJS “server”. Start off by declaring variables that we’ll use in our process.

This first part of the file imports the packages that are required, all the ones we talked about and then imports the settings files that I mentioned. The settings files look like the following:

settings.json

projectname The project name should be unique as it’s used by the gulp-cache to generate the cache for the project, reuse will potentially have two projects using the same cache and the result will be unpredictable at best.
srcFiles This is an array of Globs that should be watched, see the primer for how to form the pattern matches for folder structure.
siteCollURL The URL for the site collection you want to upload the files to.
destFolder The site collection relative path to the library / folder you want the files uploaded to.

settings_security.json

rootFolder The root of the project relative to the drive letter (i.e. C:). A problem I’ve had in the past is that the rootFolder path is case sensitive.
username Either the O365 account (in the form of an email address) or for on premises environments the username below would take the form domain\username.
pwd The password for the username provided.

You now have two options. You can either upload the files maintaining the relative folder structure of your source into the destination or you can flatten the relationship which puts all the files in the destination’s root. Below I have two functions one maintains the folder structure and the other flattens it. The idea here is that you get a listing of the files then pass them through the cache, then map the files that changed into a function call that passes each individual file to spsave which then uploads the file to the library. Because I normally use the more advanced version that I’ll be showing in part 4, which pretty much creates only 1 file, I almost always use the flatten option.

Finally, we have the watch function that either calls the folder version or the flat version of our copyToSharePoint function. Obviously if you have a preference you don’t need to keep both of these functions in your gulpfile, I’m just putting them both here for reference.

Now if you run “gulp watchFolder” or “gulp watchFlat” from the command interface node will run your gulp package which will keep watch on your folder structure and when you save a file spsave will upload it to your SharePoint site and if you refresh the page you’re updated code will be available.

Using WebDAV to upload your files into SharePoint – SharePoint 2010 and below

Before we had OneDrive in all its current synchronization support glory it was very common for organizations to use WebDAV to give their users access to their SharePoint libraries via the windows explorer. We can exploit that functionality within our development environment and the fact that Nodejs allows us to interact with the full file system since it’s running on our computer. Basically, anything we have access to we can access via our gulpfile. That said we’ll need slightly different tools for the process since instead of using a http post to upload our files to the SharePoint library we’re going to copy them to the server via WebDAV.

First let’s modify our package.json file to include a new tool called vinyl-fs. This tool allows us to map files from a source location to a destination location. There are a lot of options that you can use with this package, and if you want to dig further in please take a look at the documentation. To modify the package.json file you can either execute the ‘npm i vinyl-fs –save-dev’ or modify your package.json file directly to add the file to your devDependencies section and then run ‘npm i’ which will install any missing packages. So in my package.json file devDependencies section I have added:

Now we need to modify our gulpfile.js to include a reference to vinyl-fs and to add a new version of our copyToSharePoint function that is specific for this legacy version. First add a variable at the top of the file to reference vinyl-fs

Then add the following function which supports the WebDav path.

And finally, either modify or add an additional watch task that will utilize the legacy copyToSharePoint function

Before you run your watch make sure to update your settings.json file to put the WebDav path as your destFolder. You can optionally map the WebDav folder as a drive using ‘net use’ and then reference the drive letter, just make sure to use forward slashes not backslashes.

Up Next

In the next installment I’ll be covering how to implement a basic version of Webpack into your environment which will bundle all your html, css, and js into one file.

Series Navigation<< Conquer your dev toolchain in ‘Classic’ SharePoint – Part 1Conquer your dev toolchain in ‘Classic’ SharePoint – Part 3 >>

    Julie has been building software on primarily the Microsoft platform for over 20 years. With a degree in Electrical Engineering specializing in microprocessor system design from Worcester Polytechnic Institute she came at software initially from a very low level but quickly realized a love and aptitude for developing rich user experiences and solutions. She worked her way up through internal IT ranks reaching a Director of IT position before realizing more impact could be made in consulting and really focused in on the SharePoint platform in 2007. Since then her focus has been on the SharePoint platform, Office 365, Azure, and client side development. She's the co-author of the Widget Wrangler JavaScript library and continues to try and help innovate SharePoint and Office 365 solutions for her clients. She is a 2017 recipient of the Microsoft MVP award for Office Servers and Services.


    4 comments
    1. […] Conquer your dev toolchain in ‘Classic’ SharePoint – Part 2 (Veni, Vidi, Vici) In the first post in this series I discussed some of the benefits of formalizing your client-side development process and then a bit about starting the process of tooling up. A common scenario to develop our own client-side solutions in SharePoint is to point a Content Editor (CEWP) or Script Editor (SEWP) web part at our custom html, css, and js files that are sitting in a document library somewhere in our environment. In this post I want to dive into the most basic implementations of a development toolchain that will automatically deploy those files into a location in SharePoint. This does not mean, and in most cases, should not mean, “production”. What it means is that while you’re developing your code any changes you make will be automatically uploaded to a location that you already have your CEWP or SEWP pointed to, that way when you refresh the page your custom solution will refresh with the latest version of your code. To accomplish this, we’re going to use Gulp which is a JavaScript based task runner similar to the build process that exists in Visual Studio. […]

    2. I learned a lot with this one. Had no idea you could save to SharePoint from gulp. Mind blowing. One question though, can you do this without storing a password in clear text? I see the dependency has an option that uses the client secret instead, just not sure if that applies to the gulp method you show: https://github.com/s-KaiNet/node-sp-auth/wiki/SharePoint-Online-addin-only-authentication

      • Glad you liked it! As you point out spsave takes a dependency on node-sp-auth, all it wants for it's second parameter is a credential object that matches what that library creates... so if you want to create it outside the spsave process you can. What I'm not sure I understand is why storing the client secret as plain text in my settings_security.json file would be any better than saving my password there given that it's on my machine and I'm not sharing said file with anyone? To me, this wouldn't be worth the hassle. If you really wanted to make it more secure you'd probably create a credential object when the watch starts that prompts you for the username and password via the no authentication option. Then you can use that credential object for the life of the watch.

        • Creating a credential object when the watch starts sounds like a great idea. If there's a chance a password could be used to do harm, I wouldn't store it in plain text.

    Leave a comment

    Leave a Reply

    %d bloggers like this: