Keras

Keras: Deep Learning for Humans

By Marcel Bernard, Connor Bracewell, Christopher Brett, Andrew Stocks, and Jordan Wang

1. Introduction

The recent growth of industry interest in machine learning has led to the development of a number of powerful deep learning libraries, such as TensorFlow, Theano, and CNTK. While these tools are powerful, they also tend to reflect the complexity inherent to the machine learning field, giving them a potentially steep learning curve for newcomers.

Keras has emerged as a tool that allows users to leverage the power of these machine learning libraries, while at the same time reducing user-facing complexity and encouraging rapid experimentation by providing an accessible and easy-to-understand Python interface to these technologies.

The documentation assembled in this package provides in-depth analysis of the Keras architecture, and makes an attempt to understand both the design of Keras itself, and the key motivations and drivers behind the design. The key stakeholders in the Keras project are identified, along with several of their key business goals, and the translation of those goals to architecturally significant requirements is elucidated. Two primary architectural views are provided, as derived from the Keras source code and documentation. The first of these views, a module view, describes the dependency relationships between the key static components of Keras. The second view, a component and connector view, describes the runtime components involved as part of the main training loop functionality of Keras. The analysis concludes with a look at potential instances of technical debt taken on during the development of Keras.

3. Stakeholders

Machine learning, as a field, is of interest to a range of different stakeholders, from major corporations which use it in their products, to tinkerers and students who are interested in potentially novel applications. As such, the first step in understanding Keras' architecture is to record the different stakeholders who are invested in different aspects of its design.

Stakeholder Type	Description	Specific Parties
Acquirers	Oversee the procurement of the system or product	Project managers, IT staff, individual developers: anyone wanting to obtain a copy of the software may do so directly, for free, from the Python Package Index (PyPI) or GitHub.
Assessors	Oversee the system’s conformance to standards and legal regulation	see Developers: An open-source component in a newly-developing area, Keras is mostly free of compliance issues. However, the open-source developers are still responsible for maintaining the quality of the API and ensuring that such upstream changes do not create issues for users.
Communicators	Explain the system to other stakeholders via its documentation and training materials	Open-source developers, teachers/professors (e.g. in an A.I. course), tutorial websites: Developer-produced documentation for installing and running the software is provided alongside the source code in GitHub; users are encouraged to contribute concise and idiomatic sample code for others to examine [1]; numerous online articles and tutorials give detailed explanations of Keras' features.
Developers	Construct and deploy the system from specifications (or lead the teams that do this)	Lead developer François Chollet, open-source developers (via GitHub), corporate development teams: Originally developed as part of a research project with primary author François Chollet, Keras has since expanded in popularity and scope, with over 600 open-source contributors on GitHub. Additionally, corporate development teams have contributed to support Keras and integrate it with their own software, with development primarily backed by Google [2].
Maintainers	Manage the evolution of the system once it is operational	Keras mailing list, also see developers: New feature ideas and API suggestions are first submitted to the Keras mailing list, where the Keras developer community can determine if and how the proposed changes should be made [1].
Production Engineers	Design, deploy, and manage the hardware and software environments in which the system will be built, tested, and run	Corporate development teams, also see users/developers: In general, developers are responsible for maintaining their own build/test environments, while users are responsible for maintaining their own environments for running the software. Certain corporate development teams are responsible for maintaining software backends which integrate Keras with different underlying deep learning frameworks [2].
Suppliers	Build and/or supply the hardware, software, or infrastructure on which the system will run	See production engineers: Generally, no special-purpose hardware or software environments are required to run the software; developers and users can build their own environments using whichever products or systems they prefer.
Support Staff	Provide support to users for the product or system when it is running	See developers: The Keras development community provides support through the Keras Slack, GitHub, Google+ Group, and official documentation [3].
System Administrators	Run the system once it has been deployed	See users: Each user manages their own system after deployment.
Testers	Test the system to ensure that it is suitable for use	See developers: When submitting code changes, developers are required to run an existing test suite across all available backends and produce new tests for updated behaviour [1].
Users	Define the system’s functionality and ultimately make use of it	Scientists, Machine Learning Researchers, Students, Industry Users, Hobbyists: Keras is designed to "make deep learning as accessible as possible, without dumbing down anything" [4]. As such it is of interest to a wide range of users, from hobbyists and students being introduced to the concepts to large corporations such as Google, Microsoft, and Amazon [2].

4. Business Goals

Once the stakeholders have been recorded, we can consider their interests in Keras in order to compile a list of the most important business goals for the project. The following concepts are critical to one or more of Keras' stakeholders, and serve as underlying principles of the whole project.

1. "[M]aking [deep learning] technologies available to as many people as possible, that has become the number one design goal of Keras" [4]

   • Keras has been widely adopted in both industry and research [5]

   • It was the second most mentioned deep learning framework in scientific papers as of October 2017 [5]

   • Keras is implemented in Python for its popularity, ease of use, and surrounding software ecosystem [4]

2. "[Try] to make deep learning as accessible as possible, without dumbing down anything" [4]

   • The Keras API was designed specifically with user experience in mind [6]

   • Keras offers performance and features that are comparable or equal to using the underlying backends directly, with Python syntax that is "compact, easier to debug, and allows for ease of extensibility" compared to typical deep learning configuration

3. Maintain full flexibility in allowing the use of lower-level maching learning libraries such as Tensorflow [2]

   • Gives the user freedom to change between backends of their choice, including TensorFlow, Theano, and CNTK [7]

   • Simplifies adoption of other libraries, such as the upcoming backend for MXNet[2]

   • The Keras API is also packaged directly in TensorFlow [8]

   • To enable fast experimentation. This ties to both the speed of developing models (programming time)[3], as well as the speed of running them (evidenced by the integration of well-optimized backends[7] and GPU support[3]).

5. Architecturally Significant Requirements

Once the business goals have been identified, we aim to map them into quantifiable requirements which can serve as evaluation points for the project implementation. These requirements are organized into a utility tree according to the most relevant quality attributes, and further specified in a set of scenarios, two of we explore in full detail.

Quality Attribute	ASR	Description
Reliability	Error Recovery	If an error occurs in Keras while training or running a model, the user should be able to continue with minimal disruption. Ideally this means the user should be able to inspect and correct the error, then continue precisely where their processing left off; practically, some or all of their progress may be lost, but the user should be able to restart Keras and begin processing again without any intermediate steps.
Usability	Configuration Readability	Users should be able to read and interpret the Keras syntax for model configuration without constantly referencing the documentation. Functionality should be apparent to an experienced user at a glance.
Usability	Configuration Compactness	Keras should allow users to configure their models using fewer lines of code by abstracting away technical details that aren't relevant to the model.
Usability	Configuration Reusability	Keras configuration should be easily shareable between projects and be easily modularizable. Configuration should be easily modifiable so users can iterate on their models.
Performance	Training / Running Performance	Keras should be able to train/run a model on a given data set using comparable time and resources to the equivalent model written directly in one of its deep learning backends.
Performance	Training / Running Scalability	Keras should be able to scale its performance when large amounts of computing resources are available (via parallelization or other means) in a way that is proportional to the scalability of its deep learning backends.
Portability	Backend Portability	Keras models should be conveniently usable with any of the supported deep learning backends, so users can select the best one for their uses. Minimal (if any) modifications should be required to model configuration when changing backends. Users should easily be able to install a new backend and reconfigure Keras to use it for their models.
Portability	Hardware Portability	Like the deep learning backends it supports, Keras should be able to train/run on a variety of hardware, including CPUs, GPUs, and other specialized processors.
Extensibility	Backend Integration	Keras should be open for the addition of integration with future deep learning backends. Development teams for these backends should be able to collaborate with Keras developers to produce such integrations within reasonable time and effort constraints.
Extensibility	Modelling Integration	Keras syntax should be open to the addition of new techniques for modelling in deep learning. It should be possible for new behaviour to be added to the syntax in a concise way without adversely affecting existing models.

5.1 Utility Tree

Utility tree

5.2 Scenario Tables

Aspect	Details
Scenario Name	Model Training
Business Goals	4: To enable fast experimentation, with respect to both model development speed (in developer hours) and run-time speed (in CPU time). This is evidenced by the integration of highly-optimized backends and technologies like GPU support.
Quality Attributes	Performance
Stimulus	A user initiates the training of a model.
Stimulus Source	Any Keras user
Response	The model undergoes training according to the parameters set by the user.
Response Measure	Training takes no more than 120% of the time required to train an equivalent model (with the same training data) built directly in any underlying backend framework (TensorFlow, Theano, or CNTK).

Aspect	Details
Scenario Name	Create A New Model
Business Goals	2: To make deep learning as accessible as possible, without dumbing down anything.
Quality Attributes	Time
Stimulus	A user wants to create a new model.
Stimulus Source	The user responsible for creating the new model.
Response	A new model is created based on the users needs.
Response Measure	Creating a new model should be able to be completed by users with little to no prior experience using Keras and having only read the documentation.

Aspect	Details
Scenario Name	Change Backend
Business Goals	3. Maintain full flexibility in allowing the use of lower-level machine learning libraries such as Tensorflow
Quality Attributes	Time
Stimulus	A user wants to change the backend.
Stimulus Source	The user responsible for choosing the backend.
Response	Keras is reconfigured to use a different backend.
Response Measure	Reconfiguring Keras to use a different backend should take less than 5 hours to complete.

6. Module View

6.1 Main Diagram

Module view

6.2 Element Catalogue

6.2.1 Elements

Backend

This module is translates functionality from the various deep learning backends (TensorFlow, Theano, CNTK) to a common interface. It acts as a layer of abstraction between Keras and the backends, allowing higher level implementations within Keras to avoid dependencies on particular details of each backend's implementation.

Preprocessing

This module provides functionality for preprocessing of the image, text, and sequence data used as input to models.

Models

Defines the sequential model as the primary representation of a neural network, consisting of a linear set of connected layers. This module provides the following functionality related to models:

• Building and modifying

• Training

• Evaluating

• Performing predictions

• Cloning

• Saving to file

• Loading from file

Layers

This module is used for building and modifying the different types of layers that make up a sequential model. It consists of a number of various layer implementations.

Initializers

Provides functions for different edge weight initialization strategies for models.

Optimizers

This module provides several implementations of optimization algorithms used by models during training.

Metrics

Contains implementations of several accuracy measurement functions for predictions output by models.

Losses

Contains implementations of common loss functions optimized against during model training.

Regulizers

Provides classes for L1 and L2 regularization techniques applied during model training.

Engine

Contains definitions for base classes of layers and compositions of layers (networks). These class definitions define base implementations for:

• General layers

• Input layers

• Graph nodes

• Tensors

• Containers (composition of layers)

Constraints

Defines classes for constraining the range of trained weights in models.

Callbacks

Provides callback functions that can be passed to a model prior to training. They will be called during various stages of model training, such as before or after training epochs, between training batches, and before or after the training phase. In addition to customizable lambda callbacks, some of the default callback functions provide the following functionality:

• Logging

• Early stopping

• Model checkpointing (saving)

• Learning rate modification

• Tensorboard integrtion

Activations

Provides models with common activation functions, such as softmax, relu, sigmoid, and tanh.

6.2.2 Relations

There exist a number of very key relationships present between the elements of the descibed deep learning backends and the Keras API structural elements.

The backend is used by almost all other modules to interface with the underlying machine learning backends. It essentially makes up the basic operations that are combined to build the other modules.

The preprocessor takes in the various machine learning inputs and changes them such that they are easier for Keras to use. The preprocessor feeds its data directly into the low level backend for redirection to other components in Keras.

The engine comprises the majority of the code base for the software and comprises the largest amount of functionality. In principle it takes input in the form of models and layers from the neural network. It then processes the data through the generation of many callback functions that determine the behaviour of the machine learning network.

Optimizers are present in the architecture to primarily increase execution throughput from the neural network models. these optimizers take models and engine input and output their data directly into the backend through the common backend interface mechanism.

Metrics are generated to evaluate the models running on top of the Keras API. These take input from the execution engine and output generated losses and communocation to the backend for use in runtime decisions.

Contraints are generated by the layer creation routines and dramatically speed up execution by feeding their properties into the backend and heuristically reducing computation requirements.

Similar to constraints, the regularizers are generated when the layers of the neural network are and serve to feed into the backend to guide the neural network towards the desired output using heuristics.

Activations are used to model the artificial neurons in the convolutional networks. They take layers as input and produce output that is fed directly into the engine for processing on top of the main Keras API.

6.3 Context Diagram

Context diagram

6.4 Behavior Diagrams

6.4.1 Building Sequential Model

Behavior Diagram: Building Sequential Model

This behavior diagram shows the building of a Keras model from intializing a new model object through adding layers and testing data. It shows the interaction between the Model, Layer, and Backend modules, with the Backend being where Keras touches the actual machine learning libraries.

6.4.2 Saving Model to HDF5

Behavior Diagram: Saving Model to HDF5

This behavior diagram shows the behavior of a Keras model when saving its configuration and current layer and optimizer weights to an HDF5 file, which allows for the efficient storage of the large amount of data that makes up each model.

Note how the model accesses the configuration of its component loss function, metrics, optimizer, and layers (which in turn access the configuration of their component initializers, regularizers, activations, and constraints) in order to collect all the data that makes up the model.

All of the information is collected by the model and sent to the external h5py dependency which organizes and stores it in the final HDF5 file.

6.5 Module Organization Rationale

For a user of Keras, the project's module architecture is designed to separate a Keras model into familiar and easily-combined units.

Each piece of behavior in a model is implemented as an independent module: metrics, optimizers, loss functions, and layers, as well as the regularizers, initializers, constraints, and activations for each layer. There are a variety of descriptively-named and easy-to-configure implementations available for each component, each adhering to the same interfaces.

This allows the user to easily implement the desired behavior for their model; they simply identify the relevant module implementations, configure them, and connect them in the prescribed way. Thus the configuration and connections of each component in the model can be understood by reading just a few lines of code. These elements of the design address the usability qualities of configuration, readability, and compactness.

Because each component uses the same interfaces for each of its implementations, so it is also easy for the user to reconfigure their model; they may simply remove the code for an existing component and replace it with code for the updated version. Separating components into separate modules also allows them to be shared among completely separate code bases, allowing for sharing and re-use between projects. These elements of the design address the usability quality of configuration reusability.

For a developer of Keras, the project's module architecture is designed to isolate technical complexity without sacrificing performance.

The backend module interfaces with nearly every other module in the project, providing a layer of abstraction that separates the Keras implementation from that of the underlying machine learning backends. It provides a variety of primitive operations which can be performed on data being processed by each model; these primitive operations are then combined to produce nearly all the behavior of the other modules.

When developing a new feature for Keras, such as a new type of layer, the developer can use the backend module's primitive operations to produce the desired behavior without writing any additional backend code. The developer can rely on the backend module to provide the appropriate behavior for whichever supported backend is being used, even if the supported backends are changed. Compared to writing separate implementations of each behavior for each backend, this offers a drastic reduction in the technical complexity of implementation, and in the potential for programming errors or incompatibilities. Thus these elements of the design address the extensibility attributes of the project by simplifying the implementation of new machine learning features, as well as the portability attributes by ensuring behavior will work correctly regardless of the backend being used.

Similarly, when a new backend is being developed for Keras, the backend module provides the developers with a simplified set of operations to support. Rather than having to re-implement behavior for the wide range of already-existing components, the developers can simply implement the interface of the backend module, and the existing behavior should perform as expected. This aspect of the design further addresses the extensibility attributes of the project by simplifying the production and adoption of new backend implementations.

Finally, the backend module is able to defer most computations directly to the underlying backend, which allows Keras to take immediate advantage of their advanced optimization, minimizing the performance penalty of using Keras for a model. The backend module encourages developers to minimize the amount of additional processing performed within Keras in favor of using the underlying backends. In many cases there is nearly no additional processing performed when the backend module is utilized. This element of the design addresses the performance attributes of the project by minimizing performance differences between Keras and the underlying backends.

7. Component and Connector View

7.1 Main Presentation

Component and Connector View

7.2 Element Catalog

7.2.1 Elements

Training Data

This is the input training data to a model. It typically consists of a set of observed training instances, each of which may have many attributes. Each observed training instance will also have an accompanying ground truth label, which is used to inform the model of performance following each training loop iteration. The training instances usually consist of an n-dimensional array, while the ground truth labels consist of a one dimensional array equivalent to the training matrix in the first dimension. Note that the training instance and label arrays are always converted to NumPy arrays prior to model training.

Model Configuration Data

This configuration data consists of the data required to configure the parameters of a Keras model. This data may include:

• The gradient descent optimizer to use

• The loss function used by the optimizer

• Metrics to be recorded by the model, (e.g. accuracy, mean-squared error, as well as user-defined metrics )

  See the Module View for more information on these elements.

Preprocessing

Depending on the form of the of input data, it may need to go through an optional preprocessing phase. Keras provides built-in preprocessing pipelines for image, sequence, and text data. Following preprocessing, the training instances and labels will always be in the form of n-dimensional NumPy arrays.

Model Compilation

Compilation prepares a model for training, based on the given compilation parameters, as well as the properties of the various layers of which the model is composed. The tasks carried out during compilation include the following:

• Fetching backend optimizer

• Preparing backend input and output placeholder tensors

• Initializing custom sample weightings

• Setting initial input and output weights

• Calculating initial loss function

• Calculating initial gradients

• Calculating initial metrics

Training Phase

The elements contained within this phase describe the relevant run-time structures present during model training.

Model

A model consists of a number of layers assembled during model creation. Models can be composed arbitrarily to create larger models, as long as their inputs and outputs match in shape. A model stores the underlying backend computation graph and acts as an interface for interacting with it (i.e. querying edge weights, loss values, etc.).

Layer

Layers make up the basic building blocks of models. Similar to models, layers can also be composed arbitrarily, so long as their inputs and outputs share the same shape. The set of layers within a model ultimately determines what the backend computation graph will consist of.

Backend Tensors

When training begins, a set of backend tensors describing the computations layed out by the model are passed to the backend. These tensors describe the layout of the underlying backend computation graph.

Computation Graph

This is where the actual training computations are carried out by the backend. Model edge weights and gradients are stored here.

Training Batch NumPy Arrays

During training, the Keras model passes batches of training data to the backend. This data will be passed through the backend computation graph to inform gradient descent updates on each training iteration.

Training metrics

Following each training iteration, training metrics such as loss and accuracy are passed back up to the Keras model, where they may be used by predefined callback functions prior to the next training iteration.

Trained model

On completion of training, the original model will now store the fully trained backend computation graph. The model will serve as an interface to the trained backend graph to allow for prediction and evaluation tasks.

7.2.2 Relations

The two primary relationships that require more elaboration are the relationship of model configuration and training data to model training, and of the Keras model to the backend.

Model configuration and training data - Keras model

Keras model configuration and compilation precedes the addition of training data. During configuration and compilation, the shape and type of training data required by the model is determined. Following compilation, preprocessed training data can then be processed by the model.

Keras model - backend

The Keras model and backend interact through several pathways. The model description in the Keras model is passed through to the backend and translated into a computation graph. The Keras model can then interact with the underlying computation graph, allowing for inspection and modification of edge weights, optimizers, and losses. Training data batches are also passed between the Keras model and backend during training iterations.

7.2.3 Interfaces

Model Fitness Interface:

Syntax:

model.fit((training instance object),(label objects), (batch size (integer)), (epocs (integer)), (validation data objects))

Semantics:

Using this interface facilitates the process of inputting a training data instance into a model which is then evaluated to produce a later returned fitness value for use in the Keras system. This interface can only be used on data that is formed in such a way that it is recognizable by the TensorFlow library. When the data is input into this interface it lacks the fitness metadata that is needed. After, the model is returned with relational data on the fitness of the tensorflow network included. This data can be used to refine the neural network in other parts of the Keras system.

Data Types

The interface takes input data in the form of various data types. The main input is a neural network description which consists primarily in array-like tables with transition function descriptions with added weights. The batch size and epocs are in integer format and correspond to a whole number counter input for each field. Finally, the validation objects consist mostly of a list data structures designed to be applied to the tensor network for fitness validation.

Error Handling

The interface does no error checking and will fail at run-time as a run-time exception in the tensor network code. The default behavior is to return an error at the tensor flow level and "revert" the changes made to the Keras data structures. Keras will subsequently ignore the returned error and will move on to the next function which may or may not fail depending on run-time data conditions.

Variability

The interface may change slightly depending on operating conditions. If a user passes an unreasonably large batch or epoc size then the interface may take a very large amount of time to respond as it would have a large amount of processing to contend with. This will likely cause a run-time timeout at the Keras API level and might require a job be subdivided into smaller batch sized jobs.

Quality Attributes

The performance of the interface is directly tied to the size of the training objects and the batch size. Larger jobs may negatively affect the quality attributes of time related performance by increasing run-time substantially as Keras uses pass by value instead of pass by reference style memory design. This will require large data copies at the system level and may become more apparent as more validation is required to generate a model fitness metric.

Rationale

The model fitness interface was designed in this way so a user can make a single call and receive fitness data on the tensor network object. This effectively abstracts the user away from the many low level details of calculating such a statistic and makes development with such an interface far more simple and easy to use than the traditional TensorFlow only way of obtaining the same results and object changes.

Usage Guide

To use the interface observe the following example:

model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, validation_data=(x_test, y_test))

Model Compilation Interface:

Syntax:

model.compile((loss object), (optimizer object), (metrics object))

Semantics:

Using this interface facilitates the process of compiling a tensor training model from loss, optimizer, and metric objects. This model is used by the Keras system for use in model creation. This interface is only able to use data that is in the form tensor flow is able to use, mostly specially formed tables and functions. When the data is input to the interface it is unusable by TensorFlow directly. When a model is returned it is then usable directly in TensorFlow functions for machine learning and evaluation. The model data structure produced is designed to be friendly with the majority of NumPy functions in the the Keras system.

Data Types

The model compiler interface takes three primary objects as input data. Firstly, the loss object takes the form of a priority based linked list which holds the results of the loss function output in the order of network priority. Secondly, the optimizer object is a function that also contains a set of values that are used in the tensor optimization and construction function and is used to train the created network and generate the network data structure. Lastly, the metrics object is a structure that holds data about the created model such as its batch size.

Error Handling

This interface does checks on the input objects to make sure they meet the strict specifications of the tensor backend. If an error is generated, execution is stopped and an error object is generated and propagated to the user for run-time error feedback. There is however no check for data that is nonsensical, therefore errors of this type are only detectable through thorough debugging from the user.

Variability

This interface can vary in the underlying constructs and optimizations it performs when compiling a model. Data structures generated for the created model can be size optimized to allow for quicker processing in TensorFlow. This allows for compiled models to perform in an idealistic manner in most cases.

Quality Attributes

The performance of the model compiler is directly related to the size of the model to be compiled. The underlying compiler implementation uses data very effectively and is very well optimized to generate a model in a short amount of time. This compile time is insignificant in relation to the overall run-time so does not negatively affect the quality attribute goals of the Keras framework.

Rationale

The model compiler interface was implemented in its current form to make the act of instantiating a model as simple as possible. With as few as three data objects an entire TensorFlow model can be created and used immediately after. This makes developing with the Keras framework significantly easier for developers and reduces development time required.

Usage Guide

To use the interface observe the following example:

model.compile(loss=keras.losses.categorical_crossentrop, optimizer=keras.optimizers.Adadelta(), metrics=['accuracy'])

7.3 Context Diagram

Context Diagram

7.4 Behavior Diagrams

7.4.1 Building a Sequential Model

Building a Sequential Model

7.4.2 Training a Model

Training a Model

7.5 Variability Guide

A benefit of the normalization done by Keras is that typical modifications are self-contained; since most elements (i.e. model and layer configurations, input data, etc.) are converted to a standard form, the components which process them (such as the model compiler or training loop) do not require further changes. The configuration available within Keras is designed to accommodate a wide variety of use cases without modification, as demonstrated by its existing collection of implementations for each module.

As such, the structure outlined in the main presentation above is unlikely to vary substantially, as a large number of different implementations depend on (and are implemented successfully using) its current behavior. Additionally, changes which require substantial deviation from the processes are unlikely to be implemented, as this will increase the technical complexity of the project without substantial benefit over other, more streamlined additions.

i.e. Adding a new Layer type: A new Layer implementation must be added in the appropriate location. Assuming the requisite computation functions are available in the backends (which should be the case for reasonable additions), this class only needs to implement the desired configuration options and a call method which uses the backend module to perform the desired operation. With this implementation complete, the model compilation and training loop steps will be able to automatically pass this new layer type to the backend for processing, so no further implementation work is required. (See keras-contrib for numerous community-provided implementations which use this approach to add experimental new behavior)

As such, this component-and-connector view is unlikely to change significantly in either structure or internal representation, unless major refactoring work is undertaken.

7.6 Rationale

Keras aims to streamline the process of creating new machine learning models, which it does by simplifying and standardizing the data and methods used in each step of the process:

The preprocessing step is the most obvious application of this approach, allowing users to convert a variety of input data into the format expected by Keras models without having to implement their own conversions. The simple processing done in this step allows subsequent steps to put strict limitations on the type of data handled, which reduces their complexity.

Similarly, the common structure of each Keras model (outlined in the Module View) ensures every user model can be handled by the same procedures, without requiring the user (or Keras developers) to provide specific support for particular implementations.

This approach allows a wide variety of input data types and model configurations to be handled by a (relatively) small set of machine learning procedures which are supported by Keras. It helps ensure that the extensibility, portability, and maintainability of the project is maintained without compromising users' flexibility to implement a wide variety of machine learning models.

Keras also aims to achieve similar performance to its underlying machine learning backends, which it achieves by delegating its data processing to those backends whenever possible:

The initial computation-heavy step is the compilation of the model; where layers are joined into a single computation graph that captures the behavior of the model. However, this construction is handled almost entirely within the backend, after which a reference to this graph is kept within the model for further processing.

Similarly, in the training loop step, where sample data is entered into computation graph, Keras delegates nearly all processing to the backend. As is apparent from the main presentation, the passed data is not heavily processed, consisting only of the pre-processed input data and the (initial/updated) tensor weights. The return data is then heavily processed by the backend, and updated weights are returned to Keras. All that Keras is responsible for is sampling the sample data and running (typically simple) user-provided callback functions at each iteration.

With this approach, a very small portion of the processing time for each iteration is used to execute Keras code - the majority is spent by the machine learning backend to produce an updated graph for the model. The machine learning backends used by Keras are highly optimized, and Keras' backend interface is also lightweight, meaning Keras adds little performance overhead compared to running the machine learning algorithms natively, which is a primary performance characteristic for the system.

8. Code Quality & Technical Debt

8.1 Code Quality Report

8.1.1 Introduction

The Keras repository was analyzed using a SonarCloud instance (see here), which reports information about current technical issues and sources of technical debt. After configuring the instance to ignore portions of the code base which were not of interest (demo applications, data set files, etc.), SonarCloud reported:

• 2 bugs

• 0 vulnerabilities

• 254 code smells

• 2.3% code duplication

and estimated the project had about 10 developer-days of technical debt to correct. Apart from the bugs category, which was listed as a 'C' grade, the other metrics were marked as 'A' grades.

Due to the variety of testing environments required for Keras, it was not feasible to run test coverage reports; however by examining Keras' Travis CI build logs (see here), it can be estimated that coverage is at or above 90%.

Examining the bugs reported by SonarCloud indicated that they were easy-to-fix issues (assignment of a variable to itself in both cases) rather than major deficiencies, so the remainder of the evaluation focused on examination of the code smells and duplication. SonarCloud further separates each of these into a variety of categories, which are examined in-depth below.

8.1.2 Code Quality Issues

Naming Issues

SonarCloud identified 17 cases where variable or function names violated capitalization or other naming conventions. Most of these issues could be resolved in a few minutes by carefully renaming each variable or function and its usages. This would improve the readability of the code by making it easier to differentiate variable, function, and class names, making the code easier for developers to maintain. All of these issues are of minimal importance and are simple to correct.

Commented-Out Code Issues

SonarCloud reported 12 cases where blocks of code were commented out. However, these issues are actually the result of technically-oriented comments which are being interpreted as code by SonarCloud. If desired, these comments could be rewritten to prevent them from being flagged, or else ignored.

Local Structure Issues

SonarCloud reported 59 cases where non-optimal code structures were being used. In most cases, these issues could be corrected in a few minutes by following the suggested refactoring to improve the readability and maintainability of the code.

51 of these cases were reported for the form:

if condition #1:
     if condition #2:
         ****

Which can be refactored to:

if condition #1 and condition #2:
    ****

to condense and simplify the code, and reduce ambiguity about the possibility of other cases.

4 of these cases were reported for having repeated branches in if-else structures, for instance:

If dtype == ‘float32’:
    return np.float32 
elif dtype == ‘float64’:  
    return np.float64
else:
    return np.float32

Which could be refactored to:

If dtype == ‘float64’:
    return np.float64
else :
    return np.float32

Which simplifies the code and reduces the possibility of an error due to incorrect modification of one of the duplicate branches.

The remaining 4 cases in this category were reported for containing functionally-pointless blocks of code, such as:

if initial_state is not None:
     pass

Which should be removed to simplify the code and remove potential ambiguity about its functionality.

Cognitive Overload Issues

A majority of the issues reported by SonarCloud - 170 - relate to the cognitive complexity of the code. Each issue corresponds to a code segment which requires action to improve readability and reduce the risk of developer misunderstanding. In many cases, these segments are overly-dense, too long to read in one pass, or potentially error-prone due to unclear definitions and usages. Many of these issues would require a developer to refactor the corresponding segment and reduce it into easier-to-understand components in order to make it more comprehensible, which could take anywhere from a few minutes to several hours for some of the more substantial instances. For instance, the most major identified case, Model.compile in engine/training.py, is over 400 lines long and includes numerous branching statements and calls to other functions. This cognitive overload issue means that this piece of application-critical code is extremely challenging for most developers to understand, and makes it hard to maintain or update.

These cognitive overload issues can be further broken down into a few categories:

High Number of Parameters

The code quality report identified 64 cases where a function had more than the allowable number of parameters (7). While this number of parameters might be necessary in some cases, it also makes reading and writing these function calls harder for a developer, and makes it more challenging to maintain and update the behavior of each function. Many of these cases could be improved by separating the function's behavior into multiple smaller functions, or by converting parameters into compound data types which are easier for developers to understand.

For instance:

related to a function having too many input parameters. This makes the code hard to read and likely hard to track changes over time. To fix such an issue it is recommended that either a function is divided into two or more smaller functions or the number of parameters is reduced by passing in pre-processed data that eliminates other parameters. The amount of time recommended for a human to refactor each of these suggestions is roughly 20 minutes per function. These changes do not have any effect on the performance of the system. An example of this is:

def fit(
    self,
    x=None,
    y=None,
    batch_size=None,
    epochs=1,
    verbose=1,
    callbacks=None,
    validation_split=0.,
    validation_data=None,
    shuffle=True,
    class_weight=None,
    sample_weight=None,
    initial_epoch=0,
    steps_per_epoch=None,
    validation_steps=None,
    **kwargs
):

Could likely be simplified by combining related pieces of data, such as epochs, initial_epoch and steps_per_epoch, into an epoch_configuration object which is simpler for a developer to understand and pass to other functions.

High Degree of Branching:

The code quality report identified 97 cases where code paths contained too many conditional branches (usually in the case of nested if-else statements. Excessive branching and long branch paths make it hard for developers to read and understand each execution path as they have to keep track of entry conditions for each block to understand the program state, even though those blocks may be challenging to see at a glance or not visible on-screen at all. These cases could be improved by dividing cases and functionality into smaller, easier-to-understand functions to reduce the amount of nested logic, and simplifying conditional statements to make them easier to read. Many such improvements could be made in 20-30 minutes each, though the exact implementation details require careful attention from the developers.

For example, a small section of code in training.py:

if x_weight is None or len(x_weight) == 0:
    return [None for _ in output_names]
if len(output_names) == 1:
    if isinstance(x_weight, list) and len(x_weight) == 1:
        return x_weight
    if isinstance(x_weight, dict) and output_names[0] in x_weight:
        return [x_weight[output_names[0]]]
    else:
        return [x_weight]
if isinstance(x_weight, list):
    if len(x_weight) != len(output_names):
        raise ValueError(...)
    return x_weight
if isinstance(x_weight, dict):
    x_weights = []
    for name in output_names:
        x_weights.append(x_weight.get(name))
    return x_weights
else:
    raise TypeError(...)

Even in this fairly minor example, it can be challenging to determine at a glance how the current program state is being determined. Performing more appropriate nesting or separating some of the functionality into smaller auxiliary functions can help reduce the overall complexity of the code and make it more comprehensible and approachable for developers.

8.1.3 Conclusion

While the SonarCloud analysis of Keras did not uncover any major bugs or vulnerabilities (though it did flag 2 minor issues as major bugs), it did uncover a variety of smaller issues within the code base. While many of these issues are extremely minor, such as naming convention violations, and would require no more than a few minutes to resolve, there are others which have a greater impact on the readability and maintainability of the code.

In particular, reported issues with "cognitive overload" indicate that portions of the code should probably be refactored to simplify their structure and issues that might result if developers are unable to understand each segment in its entirety. The cause, effect, and possible resolution to these issues will be one focus of the next section of this report.

SonarCloud performed an adequately thorough evaluation of the code, and was able to identify issues with a level of thoroughness and consistency which would be hard for any human developer to match. While it is possible, if not likely, that a variety of issues were not captured in its analysis, it has helped to identify a number of simple errors which could easily be resolved to improve the quality of the project.

8.2 Potential Technical Debt

8.2.1 Introduction

To identify potential sources of technical debt within Keras, the automated analysis above was combined with a manual review of the current Keras code base. In addition to looking for common issues of readability and coding practice like those in the automated review, the reviewers also compared the current implementation to the architecture design outline produced in Milestones 3 and 4 to identify areas in which the design might have been compromised as a trade-off for other benefits.

In particular, the reviewers found two areas of potential technical debt to focus on, one identified by the automated analysis, and another that only a manual reviewer with reference to the design documentation could notice:

The reviewers agree and extend upon some of the analysis performed by the SonarCloud instance, particularly in regards to the cognitive overload created by some especially-lengthy and -complex code segments and source files. The reviewers also suggest some possible causes for these issues and their continued existence.

In addition, the reviewers identified several instances where backend code was being conditionally evaluated and executed outside the backend interface module, usually as a workaround for particular behavior of a specific supported backend. While not necessarily an issue, these inclusions go against the high-level design of the project and could result in differing behavior or hard-to-find bugs on different backends, especially as the number and complexity of supported backends considers to increase. The reviewers examine how and why these workarounds were implemented, and suggest an approach for correcting the issues.

8.2.2 Length and Complexity Issues

A number of Keras' key code segments and files have excessive length and complexity, potentially making them difficult to understand and maintain.

For instance, the Model.compile function appears to be far more complex and monolithic than necessary; rather than separating the function into independent subroutines, logical regions are delineated via comments, which can easily be missed and don't serve to fully isolate parts of the code from one another (see the following lines in the training.py file: 609, 651, 684, 735, and 816). Compared to separating the code into different functions, this approach offers less clarity about the developers' intentions and is more vulnerable to becoming stale or potentially misleading.

As another example, the _fit_loop function in training.py has several complexity issues that may affect its future maintainability: it is over 140 lines long (excluding comments and parameters) and contains deeply nested if-else statements with complex control flow. This means any developer aiming to understand and update its behavior needs to read, understand, and consider each statement and branch in the entire function, which makes implementing and testing changes more difficult than necessary. This function is a core part of Keras' functionality, as it is the primary code path for the model training loop. Therefore its correctness and maintainability is of high importance to the project; however the cost of changing it is also far higher than necessary, making it a particularly notable piece of possible technical debt within Keras.

It is possible or likely that such implementations are the result of multiple development, maintenance, and improvement iterations without the inclusion of appropriate effort for refactoring and re-evaluating of the project code. As new behavior was added, it was simply appended to the previous without consideration for the overall quality of the source files in question, and since such additions are likely to be small, the complexity increased linearly over time.

Functions like the two mentioned above are more difficult to understand and more difficult to test in fine-grained detail than functions which are broken down into components with appropriate complexity. Since each of these functions are vital to the primary Keras workflow, there is an inherent technical cost to maintaining them in their current state. While it is possible that improvements to their functionality could improve many qualities of the project, their importance and the complexity of understanding and correctly changing them in their current state may prevent such improvements from being made.

If possible, a concerted effort should be made to re-evaluate the structure of these complicated segments of code and refactor them into a form that is both easier for developers to understand, and more convenient to maintain, refactor, and improve in the future.

8.2.3 Backend-Conditional Code Paths

Several Keras modules have conditional behavior which depends upon the current backend and/or directly accesses that backend's functionality, in contradiction with the module view derived for this project. Such modules increase the complexity of the implementation and may create hard-to-fix backend-specific issues.

In the current version of Keras there are 87 segments of code which may be run depending on the current backend. Typically, these take the form of a block contained within an if K.backend() == 'particular_backend': statement, and include code which either must be run for a particular backend, or which can only be run on a particular backend.

There are several reasons why these segments are implemented in this way; for instance, some features are only available on a particular backend, such as multi-GPU models on TensorFlow, or may be required to prevent errors due to the behavior of a particular backend, such as this noted workaround for the behavior of the CNTK backend.

In the former case, it may simply be that these features cannot be implemented outside the given backend. These features typically produce error messages if an incorrect backend is selected, and thus are unlikely to introduce unexpected behavior or future bugs. Developers simply need to ensure that as support is added for each feature in subsequent backends, the code is updated appropriately and eventually unified to bring it in line with other implementations.

The latter case, however, could be considered a larger source of potential technical debt. In many cases, these workarounds are undocumented, and may be spread throughout multiple files within the project.

Therefore, in the event that the behavior of a particular backend changes, developers will be required to identify and correct these workarounds in addition to the main backend module offered for each backend. If one of these workarounds is not identified and corrected, it is possible that unexpected behavior may result, and though the developers might expect it to be the result of the backend module (which, according to the module view for this documentation, should be the only interface with particular backends), it will in fact be the result of this potentially hard-to-identify code.

The addition of such workarounds and the potential for them to introduce new errors will only increase as support for new backends is added, such as the upcoming MXNet backend.

To fully correct these issues, and avoid the complexity of further workarounds in the future, effort should be taken to isolate this backend-specific behavior within the individual backend module implementations. For instance, the noted CNTK workaround could be implemented as behavior within the CNTK backend, with the default behavior implemented in the other backends, so that a backend-independent function could be called and the appropriate functionality produced. While moving this behavior to each backend may require additional effort in the short term, in the long term it may prevent challenging issues and help the project implementation adhere to the high-level design.

Thankfully, many of these workarounds are the result of backend shortcomings which should be corrected in future releases, when they can be fully corrected within Keras. However, in the interim it is still in the project's best interest to spend effort preventing and correcting these workarounds so they do not become larger-scale issues further into the future.

8.3 Conclusion

As suggested by the SonarCloud evaluation, the technical debt currently held by Keras is actually quite low. Perhaps this can be expected from a project with high technical standards which is barely 2 years old (as of March 2018). However, it can probably also be attributed in part to the intentionally minimalist and carefully reviewed architecture and feature set of the project.

The technical debt issues examined above are currently potential future problems, rather than blockers, and remarkably few of the project's current GitHub issues can be tracked back to them directly. Many are also the result of inadequacies in the underlying backends, which should continue to improve as time goes on. It might also be argued that other issues are a reality of the complexity that underlies general machine learning applications, and should be considered a necessary compromise required to offer all of Keras' features.

In any case, Keras should aim to maintain or reduce their level of technical debt going into the future to ensure the project's complexity remains manageable. This can partially be managed by continuing with responsible coding practices, such as maintaining a high level of test coverage, as well as by maintaining current architecture decisions like the restrained feature set offered. However, there are always issues to be solved, and the Keras team should also remain committed to looking at solutions for possible technical issues such as those identified by the reviewers.

9. Conclusion

Overall, this documentation project served as an excellent exercise in examining, understanding, and documenting a complex, industry-quality software project. Despite challenges in capturing key aspects of a system as large and reconfigurable as Keras, we believe our team has captured and explained many concepts that are useful to current and future users and maintainers of the system.

Keras itself is a well-structured, and ultimately well-developed implementation of its key business goals. As such, it is unsurprising that its popularity among developers continues to grow. Its main issues are mostly the result of the underlying complexity of machine learning, and growing pains that are common to many machine learning projects today.

As Keras continues to mature, the variety of machine learning technologies available will continue to increase, as will user expectations for those technologies. By remaining committed to high-quality architectural design and implementation, it should continue to grow and improve into the future.

10. References

[1] Keras.io, "Contributing - Keras Documentation," 2018. [Online]. Available: https://keras.io/contributing/ [Accessed 22-Jan-2018].

[2] Keras.io, "Why use Keras - Keras Documentation," 2018. [Online]. Available: https://keras.io/why-use-keras/#why-use-keras [Accessed 22-Jan-2018].

[3] Keras.io, "Keras Documentation," 2018. [Online]. Available: https://keras.io/ [Accessed 22-Jan-2018]

[4] H. Browne-Anderson, “An Interview with François Chollet,” DataCamp Community, 18-Dec-2017. [Online]. Available: https://www.datacamp.com/community/blog/interview-francois-chollet. [Accessed: 21-Jan-2018].

[5] Keras.io, "Why use Keras - Keras Documentation," 2018. [Online]. Available: https://keras.io/why-use-keras/#keras-prioritizes-developer-experience. [Accessed: 22-Jan-2018].

[6] Blog.keras.io, "User experience design for APIs," 2018. [Online]. Available: https://blog.keras.io/user-experience-design-for-apis.html. [Accessed: 22- Jan- 2018].

[7] Keras.io, "Backend - Keras Documentation," 2018. [Online]. Available: https://keras.io/backend/. [Accessed: 22-Jan-2018].

[8] Keras.io, "Why use Keras - Keras Documentation," 2018. [Online]. Available: https://keras.io/why-use-keras/#keras-development-is-backed-by-key-companies-in-the-deep-learning-ecosystem. [Accessed: 22-Jan-2018].