Pulling Up Your Legacy App by Its Bootstraps

by Emily Stamey

© photo by Matt Stauffer

Emily Stamey

Application Developer at NC State University

developer NC State University logo

Emily Stamey

Application Developer at NC State University

developer NC State University logo

  • Our Team: 3 developers, 1 project manager, 1 UX professional
  • Maintain 70 legacy applications
  • Our applications support the business of the College of Engineering

Let's Talk Legacy

Supporting Legacy is Hard

 

It's harder to read code than to write it. This is why code reuse is so hard.

- Joel Spolsky

Why Bootstrapping is important

  • Web development involves supporting legacy applications

  • It is HARD to update a spaghetti codebase

  • It is also HARD to rewrite a large legacy application

  • Bootstrapping gives you flexibility to update your application

Let's review some terminology... (source: wikipedia)

Legacy Software

Software developed using older technologies and practices. It can be difficult to replace because of its wide use.

Often a negative term

Referencing a system as "legacy" often implies that the system is out of date or in need of replacement.

Spaghetti Code

The relationships between pieces of code are so tangled, it’s nearly impossible to add or change something without unpredictably breaking something.

Refactor

Technique for restructuring an existing body of code, altering its internal structure without changing its external behavior.

Technical Debt

A metaphor referring to the eventual consequences of any system design, software architecture or software development within a codebase.

How to Make Technical Debt

  • Leaving a codebase untouched for a long time
  • Rushed development
  • Inexperienced programmers built it
  • Multiple developers maintained it over time

Bootstrapping (software)

 

Building onto an existing system for the purpose of improvement with the least amount of sweat equity and development cost in the process.

The Project: Scholarships

Planning

Survey Your Application

  1. Talk to users of the application

  2. Study the codebase

  3. Examine the new feature requests

Survey Your Application

  1. Talk to users of the application

  2. Study the codebase

  3. Examine the new feature requests

Talk to Users of the Application

  1. Is their process consistent with the application?

  2. What are the pain points?

  3. Do they have concerns with the application?

Don’t rely on developer feedback

User Feedback: Scholarship Process

Process was inconsistent with the application

  • Language of the customer and the application was entirely different

  • Selection Committee used a spreadsheet of available scholarship money

  • Selected Candidates were added to a spreadsheet Student ID and Amount and Term of award

User Feedback: Scholarship Pain Points

High Margin of Error

Their process was exiting and re-entering the system through spreadsheets

User Feedback: Scholarship Concerns

  • Many awards were rejected by Financial Aid

  • They didn’t trust that Selection Committee was choosing the best candidates

  • The Scoring algorithm was not clear/effective

  • Multiple majors weren't allowed by our system

  • NOT ALL MONEY WAS BEING AWARDED

  • MONEY LEFT UN-GIVEN ⇒ ANGRY DONORS

Survey Your Application

  1. Talk to users of the application

  2. Study the codebase

  3. Examine the new feature requests

Study the codebase

  • Talk to past/current developers

  • Verify that key functionality does what everyone thinks it does

  • Look for entanglements

Codebase of Scholarship

  • Large App model

    • SQL queries, only slightly dynamic

    • Functions weren’t single-purpose

    • No Bounded Contexts between Students, Selection, and Foundation

Codebase of Scholarship

  • Student application data was a single row in table

    • Academic information wasn’t updated when it changed

    • Major was a single column in that row

Survey Your Application

  1. Study the codebase

  2. Talk to users of the application

  3. Examine the new feature requests

Examine New Feature Requests

  • What new features are needed?

  • How they might be implemented?

New Features: Scholarships

  • Explicit criteria matching, excluded non-matching applicants

  • Current student data, query their GPA, Major, etc at time of selection

  • Students have multiple majors

Before you start

  • Is there another application that can do what it does? Is it better?

  • Is this a worthwhile investment?

  • If so, what are the Most Valuable Features?

Include Customers/Users in Decisions

  • Explain why this work is necessary

  • Be open about errors in the application

  • Build trust from the beginning

Set Expectations

  • Customer has an open door to you

    • add work from the side

    • change processes

  • Keep the door open anyway!

Prioritize and Scope Work

  • Bootstrapping grows FAST!

  • Large projects

    • require more people to be engaged in the process

    • wear people down over time

  • Scope the work you are agreeing to do

Planning for Scholarships

Planning Work for Scholarships

Divided the application based on timeline

we didn't scope the work

Mitigate Risk

Control Risk Before Bootstrapping

  • Version control

    Stabilize the code base and preserves history

  • Development and Staging environments (w/ Fake data)

    No more developing in production!!!!

Test Everything You Need to Keep

  • Acceptance tests stabilize functionality you need

    • added hooks for testing interfaces

    • Codeception needed to view contents of the pages

  • Unit and Functional tests for everything you build

Composer packages

  • Testing packages (Codeception, PHPUnit, Mockery)
  • Twig templates
  • Pimple container
  • Illuminate database
  • Phinx for database migrations

Composer Require


{
    "name": "itecs/scholarships",
    "description": "Scholarships application for the College of Engineering.",
    "require": {
        "php": ">=5.3.3",
        "robmorgan/phinx": "*",
        "pimple/pimple": "~3.0",
       "ncsu/auth": "dev-master"
    },
    "require-dev": {
        "codeception/codeception": "2.0.*"
    }
    ...
}

Full Composer


{
    "name": "itecs/scholarships",
    "description": "Scholarships application for the College of Engineering.",
    "require": {
        "php": ">=5.3.3",
        "robmorgan/phinx": "*",
        "pimple/pimple": "~3.0",
        "symfony/http-foundation": "2.5.*",
        "illuminate/database": "4.1.*",
        "illuminate/support": "4.1.*",
        "ncsu/auth": "dev-master",
        "twig/twig": "~1.16",
        "mathiasverraes/money": "v1.3.0",
        "monolog/monolog": "~1.12",
        "mpdf/mpdf": "dev-master"
    },
    "require-dev": {
        "codeception/codeception": "2.0.*",
        "codeception/specify": "*",
        "codeception/verify": "*",
        "fzaninotto/faker": "1.4.*",
        "phpunit/phpunit": "4.1.*",
        "mockery/mockery": "0.9.*",
        "symfony/var-dumper": "~2.6",
        "squizlabs/php_codesniffer": "2.*"
    },
    "autoload": {
        "psr-4": {
            "ITECS\\Scholarships\\": [ "src/", "app/core/" ],
            "Codeception\\Module\\": "src/",
            "Tests\\Substitute\\": "tests/_helpers/"
        }
    }
}

Database Migrations: Phinx

  • Allows you to change DB across evironments

  • Gives you power to undo the change if there is a problem

$ vendor/bin/phinx create CreateUserLoginsTable

<?php
use Phinx\Migration\AbstractMigration;

class CreateUserLoginsTable extends AbstractMigration
{
    /* Example One: Change */
    public function change()
    {
        // create the table
        $table = $this->table('user_logins');
        $table->addColumn('user_id', 'integer')
        ->addColumn('created', 'datetime')
        ->create();
    }

}

$ vendor/bin/phinx migrate
$ vendor/bin/phinx rollback

<?php
use Phinx\Migration\AbstractMigration;

class CreateUserLoginsTable extends AbstractMigration
{
    /* Example Two: Up/Down with Migrate/Rollback */
    public function up()
    {
        $table = $this->table('users');
        $table->renameColumn('bio', 'biography');
    }

    public function down()
    {
        $table = $this->table('users');
        $table->renameColumn('biography', 'bio');
    }
}

$ vendor/bin/phinx migrate
$ vendor/bin/phinx rollback

Configuration file

  • DB connections

  • Base URL

  • Set paths to twig templates

  • Customize Notice messages

  • Set config variables for services

Configuration file

'/app/config.php' 


<?php
return array(
    /* base url for path in the site */
    'app' => array(
        'base_url' => sprintf('http://localhost:%s/', isset($_SERVER['SERVER_PORT'])
        ? $_SERVER['SERVER_PORT'] : ''),
        'index_page' => 'index.php/',
        'debug' => FALSE
    ),

    /* DB configuration */
    'db' => array(
        'default' => array(
            'hostname' => "local__server",
            'port' => "3306",
            'username' => "uname",
            'password' => "pword",
            'database' => "db_name_"
        )
    ),

    'twig' => array(
        'template_path' => array(
            __DIR__ . '/templates',
            __DIR__ . '/templates/dashboard',
        )
    ),

    'authorization' => array(
        'funding' => array('elstamey', 'person2'),
        'developers' => array('elstamey'),
        'coordinators' => array('person3')
    ),

    'notice' => array(
        'enabled' => false,
        'type' => 'info',
        'headline' => null,
        'message' => null,
        'syslink' => null
    ),

    'errorHandling' => array(
        'emailExceptions' => false
    ),

    'awardLetters' => array(
        'letterhead' => '_itecs/notification-samples/ncsu_coe_aa_letterhead.pdf',
        'academicYears' => '2015-2016',
        'respondByDate' => 'May 11, 2015',
        'templates' => array(
            'awardLetter' => 'layouts/letters/award-letter.twig',
            'awardEmail' => 'layouts/letters/award-email-body.twig'
        ),
        'fixedAttachments' => array(
            '_itecs/notification-samples/thank_you_template.pdf'
        ),
        'signatureImagePath' => '_uploads/tmp/deans_signature.png',
        'outputPath' => '_uploads/awardLetters/'
    ),

    'emailService' => array(
        'sendLimit' => 5,
        'retries' => 3,
        'throttleTime' => 2, // seconds
        'fromAddress' => 'Scholarships Application <engr-webmaster@ncsu.edu>',
        'replyToAddress'=>'College of Engineering <engineering@ncsu.edu>'
    )
);

?>

We passed this array into a $config variable that could be accessed as `$config['app']['base_url']`

The Good Stuff

Bounded Contexts

Events

  • Student Submitted Application

  • Budget Allocated To Scholarship

  • Award Was Given To Student

Bootstrapping: Filetree

  • new code in '/src' alongside the '/app' directory

File tree DDD File tree

Bootstrapping: Namespaces

  • '/src' is given a namespace

  • namespaces are autoloaded in composer


{
    "name": "itecs/scholarships",
    "description": "Scholarships application for the College of Engineering.",
...
    "autoload": {
        "psr-4": {
        "ITECS\\Scholarships\\": [ "src/", "app/core/" ],
        "Codeception\\Module\\": "src/",
        "Tests\\Substitute\\": "tests/_helpers/"
        }
    }
}

Bootstrapping: Connecting to the Framework

  • '/app/bindings.php' define container for new code and its dependencies

  • '/app/controllers'
    new controllers for the new functionality

  • '/app/services.php' defined and configured twig, database, et al

Dependency Injection

Dependency Injection Container

  • The Pimple Container helps us pass the Service or Object we need in the application along with all of its dependencies
  • We don't have to wait to let the dependencies resolve when the user accesses the object, we can ensure they are there ahead of time

Container

'/app/bindings.php' 


<?php

    /* example one */

    use Scholarships\Selection\Services\IlluminateDatabaseGpaService;

    $container['Scholarships\Common\Services\GpaService'] = function($c) {
        return new IlluminateDatabaseGpaService($c['database']);
    };

Container

'/app/bindings.php' 


<?php

    /* example two */

    use Scholarships\Selection\ApplicantQueryService;

    $container['Scholarships\Selection\ApplicantQueryService'] = function($c) {
        return new ApplicantQueryService(
            $c['Scholarships\StudentApplication\StudentQueryService'],
            $c['Scholarships\Common\Services\ResidenciesService'],
            $c['Scholarships\Common\Services\GpaService'],
            $c['Scholarships\Common\Services\UnmetNeedService'],
            $c['Scholarships\Common\Services\CandidateQualificationService']
        );
    };

Controllers


<?php

class Selectionnext extends BaseController
{
    public function Selectionnext()
    {
        parent::BaseController();

        $this->scholarshipRepository = $this->container['Scholarships\Selection\Scholarship\ScholarshipRepository'];
        $this->collaborationsService = $this->container['Scholarships\Selection\CollaborationsService'];
        $this->committeeService = $this->container['Scholarships\Selection\CommitteeService'];
        $this->authService = $this->container['Scholarships\IdentityAccess\AuthenticationService'];
        $this->awardsService = $this->container['Scholarships\Selection\Scholarship\AwardsService'];
        $this->events = $this->container['Scholarships\Support\Events\EventStore'];
    }

    private function getScholarshipDashboard($scholarshipId,$keepItLight=false)
    {
        $committeeMember = $this->getCommitteeMember();

        $presenter = new ScholarshipDashboardPresenter(
            $committeeMember,
            ScholarshipId::fromString($scholarshipId),
            $this->events,
            $this->scholarshipRepository,
            $this->container['Scholarships\Selection\ApplicantQueryService'],
            $keepItLight
        );

        $scholDetails = $presenter->asArrayForJson();
        $scholDetails['json_string'] = json_encode($scholDetails);

        return $scholDetails;
    }
    ?>

Base Controller


<?php

namespace ITECS\Scholarships;

use \ReflectionClass;
use \Log;

/**
 * Base Application Controller Class
 *
 */
class BaseController extends \Controller {

    const BAD_REQUEST=400;
    const FORBIDDEN=403;
    const NOT_FOUND=404;
    const METHOD_NOT_ALLOWED=405;
    const SERVER_ERROR=500;

    protected $serverResponseCodes;
    protected $authenticationService;

    /**
     * @var Pimple\Container
     */
    protected $container = null;

    /**
     * Constructor
     */
    public function BaseController()
    {
        parent::Controller();


        // In Ameyrika, BaseController Contains Container
        global $container;
        $this->container = $container;

         $this->authenticationService = $this->container['ITECS\Scholarships\IdentityAccess\AuthenticationService'];

         session_start();

         $this->setExceptionHandler();

         log_message('debug', "Base Controller Class Initialized");
     }

     /**
     * Fetch the current user id
     *
     * @return string
     */
     protected function getCurrentUserIdentity()
     {
        return $this->authenticationService->getCurrentAuthenticatedUser();
     }

     /**
     * Invoke twig to render content and immediately exit afterward
     *
     * @param String $name pathname of template to render
     * @param Array
     */
     protected function render($name, array $context = array())
     {
         // This keeps CodeIgniter from getting frisky.
         ob_end_clean();
         echo $this->container['twig']->render($name, $context);
         exit;
     }

     /**
     * Force a redirect with a flash notice
     *
     * @param string $location where to redirect to
     * @param string $title  a leading label for the notice
     * @param string $title  the main message for the notice
     * @param string $type   a (Twitter Bootstrap) alert type; success, info, warning, danger
     *
     */
     protected function redirectAwayWithFlashNotice($location, $title, $message, $type='success')
     {
         $this->setFlashNotice($title, $message, $type);
         redirect($location);
         exit;
     }

     /**
     * Push a flash notice into the session
     * Note: Simplistic. Last notice pushed wins.
     *
     * @param string $title  a leading label for the notice
     * @param string $title  the main message for the notice
     * @param string $type   a (Twitter Bootstrap) alert type; success, info, warning, danger
     */
     protected function setFlashNotice($title, $message, $type)
     {
         $_SESSION['flash_notice'] = array(
             'type' => $type,
             'title'=>$title,
             'message' => $message
        );
     }

     /**
     * Render a "Serious Error" page
     * e.g.  $this->dieWithErrors(self::METHOD_NOT_ALLOWED, array("You go now!", "Bad Wookie!", "I am Groot!") );
     * or    $this->dieWithErrors(self::BAD_REQUEST, "You not have done that.")
     * or    $this->dieWithErrors(self::FORBIDDEN)
     *
     * @param int $statusCode  one of the codes we actually use: 400 (Bad Request), 403 (Forbiddeni), 404 (Not Found), 405 (Method Not Allowed), 500 (Server Error)
     * @param mixed $messages  a string or array of error messages; optionally an integer code or other string-interpolatable object
     */
     protected function dieWithErrors($statusCode,$exception=NULL,$userMessages=NULL)
     {
     if (is_null($userMessages)) {
     $userMessages=array();
     } else if (is_string($userMessages)) {
     $userMessages=array($userMessages);
     } else if (!is_array($userMessages)) {
     $userMessages=array("An unexpected error occurred: {$userMessages}");
     }

     if (!in_array($statusCode, array(self::BAD_REQUEST, self::FORBIDDEN, self::NOT_FOUND, self::METHOD_NOT_ALLOWED, self::SERVER_ERROR))) {
     $statusCode = self::SERVER_ERROR;
     }

     $constants = new ReflectionClass('ITECS\Scholarships\BaseController');
     $this->serverResponseCodes = array_flip($constants->getConstants());

     $diagnosticMessages = array();
     if (!is_null($exception)) {
     $diagnosticMessages = array($exception->getMessage());
     if ($statusCode == self::SERVER_ERROR) {
     $this->reportUnhandledException($exception);
     }
     }

     $this->render("error/{$statusCode}.twig", array(

     'diagnosticMessages' => $diagnosticMessages,

     'userErrorMessages' => $userMessages,

     'currentUser'   => ( $statusCode == 403 ? array('name'=>'Access Denied') :  array('name'=>ucwords(strtolower(str_replace('_', ' ', $this->serverResponseCodes[$statusCode])))) ),

     'currentUrlEncoded' => $current_url = str_replace('+','%20', urlencode(sprintf(
     'Scholarships problem at %s://%s/%s',
     isset($_SERVER['HTTPS']) ? 'https' : 'http',
     $_SERVER['HTTP_HOST'],
     $_SERVER['REQUEST_URI']
     )))
     ));

     exit;
     }

     /**
     * General exception handler; Guards against direct URL access
     *
     * @param Exception $exception
     * @access private
     */
     public function handleUncaughtExceptions($exception)
     {
     $caller = debug_backtrace();
     $caller = $caller[1]['function'];

     if (!empty($caller)) {
     $this->dieWithErrors(self::NOT_FOUND);
     }

     $this->dieWithErrors(self::SERVER_ERROR,$exception);
     }


     /**
     * Set a default exception handler
     */
     protected function setExceptionHandler()
     {
     set_exception_handler(array($this, "handleUncaughtExceptions"));
     }

     /**
     * Send an email notice to the team that an unhandled exception was detected
     * @param Exception $exception
     */
     protected function reportUnhandledException($exception)
     {
     if ( $this->container['config']['errorHandling']['emailExceptions'] ) {
     $mail_body = "An unhandled exception was detected in Scholarships.";
     $mail_body .= "\n\nThe exception code was {$exception->getCode()}\n\nThe exception message was:\n{$exception->getMessage()}\n\nThe current user was '{$this->getCurrentUserIdentity()}'.\n\n";
     $mail_body .= "Request URI: {$_SERVER['REQUEST_URI']}\n\n";

     mail(
     $recipient = 'engr-webmaster@ncsu.edu',
     $subject   = '[Scholarships] Unhandled Exception on ' . date("F j, Y, g:i a"),
     $mail_body,
     $header    = "From: Scholarships Application \r\n"
     );
     }

     Log::error($exception, array('login' => $this->getCurrentUserIdentity()));
     }

     /**
     * Check if request HTTP method is POST
     *
     * @return bool
     */
     protected function requestMethodIsPost()
     {
     return $this->container['request']->getMethod() === 'POST';
     }

     /**
     * Guard against access to a controller action except through POST method
     *
     * @param mixed $message   string or array of strings indicating cause of failure
     * @access private
     */
     protected function methodMustBePost($message)
     {
     if ( !$this->requestMethodIsPost() ) {
     $this->dieWithErrors(self::METHOD_NOT_ALLOWED, NULL, $message);
     }
     }
     }

Summary: Scholarship Wins!

  • Implemented multiple majors successfully
  • Eliminated unqualified candidates
    • made the scoring easier to read
    • reduced the manual review of candidates
  • Selection process was entirely inside the application

Summary: Scholarship Wins!

  • Restored confidence in selection process!
  • Fewer awards were rejected!
  • More Scholarship Money was awarded in the application than ever before!

By May 2015: Approximately $1,074,394 Awarded

Summary: Lessons Learned

  • Tight deadlines with un-scoped work, we created technical debt that we would have to address in the next academic year

Summary: Lessons Learned

  • We learned a lot of new techniques
    • Bootstrapping Legacy Apps
    • Event Sourcing
    • Domain-Driven Design
    • Command Query Response Segregation
    • Project Management
    • A LOT!

Summary: What we Finished

  • Replaced the full Student Application
  • Replaced the Selection Process with improved functionality
  • Built Event-source distribution of scholarship money

Old Selection Interface

New Selection Interface

Planning: Year Two

  • Rollover between Academic years
  • The Scholarship CRUD
  • Beginning of the year Fund allocations
  • Increase/Decrease of Funds

University is replacing it

What now?

  • Backed up and removed all events from the previous year
  • Minor code improvements, but minimal effort
  • Just completed the second year's primary selection window
  • Gave them two years of selection until Academic Works could get up and running
  • Expecting Academic Works to come online soon.

QUESTIONS?

 

Emily Stamey

Twitter: @elstamey

Blog with Bootstrapping details: elstamey.com