The Setting of call_back for Keas Model Training and Preservation

1. Model training

fit(x=None, 
y=None, 
batch_size=None, 
epochs=1, 
verbose=1, 
callbacks=None, 
validation_split=0.0, 
validation_data=None, 
shuffle=True, 
class_weight=None, 
sample_weight=None, 
initial_epoch=0, 
steps_per_epoch=None, 
validation_steps=None)


//Parameters:
x: Training data Numpy Array (if the model has only one input), or Numpy List of arrays (if the model has multiple inputs). If the input layer in the model is named, you can also pass a dictionary that maps the input layer name to Numpy Array. If fed from a local frame tensor (for example TensorFlow Data tensor) data, x It could be None(Default).
y: Target (label) data Numpy Array (if the model has only one output), or Numpy List of arrays (if the model has multiple outputs). If the output layer in the model is named, you can also pass a dictionary that maps the output layer name to Numpy Array. If fed from a local frame tensor (for example TensorFlow Data tensor) data, y It could be None(Default).
batch_size: Integer or None. The number of samples per gradient update. If not specified, the default is 32.
epochs: Integer. The training model is iterated. One round is in the whole x and y The last iteration. Please note that with initial_epoch Together? epochs Be understood as 「Final rounds」. Models are not training anymore. epochs Wheel, but to epochs Round stop training.
verbose: 0, 1 Or 2. Log display mode. Zero = Quiet mode, 1 = Progress bar, 2 = One line per round.
callbacks: A series of keras.callbacks.Callback Example. A series of callback functions that can be used in training. 
validation_split: 0 Floating point number between 1 and 1. Proportion of training data used as validation sets. The model will separate out some validation data that will not be trained, and will evaluate the error of these validation data and any other model indicators at the end of each round. Validation data before mixing x and y The last part of the data sample.
validation_data: tuple (x_val,y_val) Or tuples (x_val,y_val,val_sample_weights), It is used to assess losses and any model metrics at the end of each round. The model will not be trained on this data. This parameter will override validation_split. 
shuffle: Boolean values (whether to shuffle data before each iteration) or strings (batch).  batch It's processing. HDF5 A special option for data restriction, which is for a batch Internal data are mixed. When steps_per_epoch wrong None This parameter is invalid.
class_weight: An optional dictionary is used to map the class index (integer) to the weight (floating point) value, and to weigh the loss function (only during training). This may help to tell the model 「More attention」Samples from underrepresented classes.
sample_weight: Selection of training samples Numpy An array of weights used to weigh the loss function (only during training). You can pass the same flat length as the input sample (1) D)Numpy Array (1 between weights and samples):1 Mapping), or in the case of sequential data, the size can be passed as (samples, sequence_length) 2 D Array to apply different weights to each time step of each sample. In this case, you should make sure that compile() Designated in sample_weight_mode="temporal". 
initial_epoch: Integer. Start training rounds (to help recover from previous training).
steps_per_epoch: Integer or None.  The total number of steps (sample batches) before declaring the completion of one round and the start of the next. Use TensorFlow Default values of input tensors such as data tensors for training None Equal to the number of samples in the data set divided by batch The size, if not determined, is 1.
validation_steps: Only when specified steps_per_epoch Only when useful. Total number of steps to verify before stopping (batch samples)
fit_generator(generator, 
steps_per_epoch=None, 
epochs=1, 
verbose=1, 
callbacks=None, 
validation_data=None, 
validation_steps=None, 
class_weight=None, 
max_queue_size=10, 
workers=1, 
use_multiprocessing=False, 
shuffle=True, 
initial_epoch=0)

Batch-by-batch training model using data generated by Python generator (or Sequence instance)
parameter

Generator: A generator, or an instance of a Sequence (keras.utils.Sequence) object, to avoid duplication of data when using multiple processes. The output of the generator should be one of the following:
    A tuple (inputs, targets)
    A tuple (inputs, targets, sample_weights).
This tuple (the single output of the generator) makes up a single batch. Therefore, all arrays in this tuple must be the same length (equal to the size of this batch). Different batches may vary in size. For example, the last batch of an epoch is often smaller than other batches if the size of the data set cannot be divisible by batch size. The generator will loop indefinitely over the data set. When running to steps_per_epoch, remember that an epoch ends.

steps_per_epoch: The total number of steps (batch samples) generated from the generator before declaring that an epoch is complete and starting the next epoch. It should usually be equal to the sample size of your data set divided by the batch size. For Sequence, it is optional: if not specified, len(generator) is used as the number of steps.

Epochs: Integer. The total number of iterations of the training model. An epoch is an iteration of the entire data provided, as defined by steps_per_epoch. Note that when used with initial_epoch, epoch should be understood as "the last round". The model does not undergo repeated iteration training given by epochs, but only until the rounds of index epochs are reached.
verbose: 0, 1 or 2. Log display mode. 0 = quiet mode, 1 = progress bar, 2 = one line per round.
Callbacks: List of instances of keras. callbacks. Callback. A series of callback functions called during training.
validation_data: It can be one of the following:
    Generator or Sequence instance of validation data
    A tuple (inputs, targets)
    A tuple (inputs, targets, sample_weights).
    Assess the loss and any model indicators at the end of each epoch. The model does not train the data.
validation_steps: validation_data is available only if it is a generator. Total number of steps (sample batches) generated by generator before stopping. For Sequence, it is optional: if not specified, len(generator) is used as the number of steps.

class_weight: An optional dictionary that maps class indexes (integers) to weighted (floating-point) values for weighted loss functions (only during training). This can be used to tell the model to "pay more attention" to samples from underrepresented classes.
Max_queue_size: Integer. Maximum size of generator queue. If not specified, max_queue_size will default to 10.
Workers: Integers. Maximum number of processes used, if process-based multithreading is used. If not specified, workers will default to 1. If 0, the generator is executed on the main thread.
Use_multiprocessing: Boolean value. If True, process-based multithreading is used. If not specified, use_multiprocessing will default to False. Note that since this implementation relies on multiple processes, you should not pass unpassable parameters to generators, because they cannot be easily passed to child processes.
shuffle: Does it disrupt batch order before each iteration? It can only be used with Sequence (keras.utils.Sequence) instances.
initial_epoch: Round of start training (helps to restore previous training)

Both fit and fit_generator functions return a History object whose History.history property records the changes of the values of loss function and other indicators with epoch. If there is a verification set, it also includes the changes of these indicators of the verification set, which can be written to the text for subsequent viewing.

2. Preserving model structure, training weight and optimizer state

keras.callbacks.ModelCheckpoint(filepath,
 monitor='val_loss', 
verbose=0, 
save_best_only=False, 
save_weights_only=False, 
mode='auto', 
period=1)

//Save the model after each training period
//Parameters:
filepath: String to save the path of the model. as epoch1.h5 perhaps epoch1.weight(save_weights_only=True)
monitor: Data monitored.
verbose: Detail mode, 0 or 1.
save_best_only: If save_best_only=True, The best model of monitored data will not be overwritten.
mode: {auto, min, max} One of them. If save_best_only=True,The decision whether to overwrite the saved file depends on the maximum or minimum value of the monitored data. about val_acc,The pattern would be max,For val_loss,Patterns need to be min,Wait. stay auto In the pattern, the direction is automatically determined from the name of the monitored data.
save_weights_only: If True,Then only the weight of the model will be preserved. (model.save_weights(filepath)), Otherwise, the whole model will be saved. (model.save(filepath)). 
period: Interval between each checkpoint (number of training rounds)

//Example:
checkpoint = ModelCheckpoint(filepath=model_weight_filepath, monitor='val_acc',
                                 verbose=0, save_best_only=True, save_weights_only=True, mode='max', period=1)
model.fit(X_train, Y_train, callbacks=[checkpoint])

3. How to interrupt the training when the verification loss is no longer reduced? Discontinue training when monitoring values no longer improve

EarlyStopping Callback function:

keras.callbacks.EarlyStopping(
monitor='val_loss', 
min_delta=0, 
patience=0, 
verbose=0, 
mode='auto',
baseline=None, 
restore_best_weights=False)

//When the number of monitored is no longer increased, the training is stopped.

//Parameters:
monitor: Data monitored.
min_delta: In the monitored data is considered to be the smallest change in the ascent, for example, less than min_delta Absolute change will be considered as no improvement.
patience: The number of training rounds without improvement will be stopped after that.
verbose: Detailed information mode, 0 or 1.
mode: {auto, min, max} One of. stay min In the model, when the monitored data stops falling, the training stops. max In the model, when the monitored data stops rising, the training stops. auto In the pattern, the direction is automatically determined from the name of the monitored data.
baseline: The baseline value of the quantity to be monitored. If the model does not show an improvement in the benchmark, the training will stop.
restore_best_weights: Whether to restore the weight of the model from the period with the best value of the number of monitored data. If False,The model weights obtained at the last step of training are used.

//Example:
earlystopping = EarlyStopping(monitor='val_acc', verbose=1, patience=3)
model.fit(X_train, Y_train, callbacks=[earlystopping])

4. Dynamic adjustment of learning rate

keras.callbacks.ReduceLROnPlateau(
monitor='val_loss', 
factor=0.1, 
patience=10, 
verbose=0, 
mode='auto', 
min_delta=0.0001, 
cooldown=0, 
min_lr=0)

//When the standard assessment stops improving, the learning rate is reduced.
//When learning stops, models always benefit from a 2-10-fold reduction in learning rate. This callback function monitors a data and when the data does not improve after a certain "patient" training round, the learning rate will be reduced.

//parameter
monitor: Data monitored.
factor: A factor that reduces the learning rate. New learning rate = Learning rate * factor
patience: If there is no improvement in the number of training rounds, then the training rate will be reduced.
verbose: Integer. 0: Quiet, 1: Update information.
mode: {auto, min, max} One of. If it is min In the model, the learning rate will be reduced if the monitored data has stopped declining. max In the model, learning plastics will be reduced if the monitored data has stopped rising; auto Patterns, directions are automatically inferred from the monitored data.
min_delta: For measuring new optimal thresholds, focus only on dramatic changes.
cooldown: After the learning rate is reduced, the number of training rounds waiting before resuming normal operation is restored.
min_lr: The lower bound of learning rate.

//Example:
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2,
                              patience=5, min_lr=0.001)
model.fit(X_train, Y_train, callbacks=[reduce_lr])

4. Tensorboard visualization

keras.callbacks.TensorBoard(
log_dir='./logs', 
histogram_freq=0, 
batch_size=32, 
write_graph=True, 
write_grads=False, 
write_images=False, 
embeddings_freq=0, 
embeddings_layer_names=None, 
embeddings_metadata=None, 
embeddings_data=None, 
update_freq='epoch')

//The callback function writes a log for Tensorboard, so that you can visualize the dynamic image of the standard evaluation of the test and training, as well as the activation histogram of different layers in the model.
//If you have installed Tensorflow using pip, you should start Tensorflow from the command line:
tensorboard --logdir=/full_path_to_your_logs

//Parameters:
log_dir: Used to save quilt TensorBoard The file name of the log file analyzed.
histogram_freq: For each layer of the model, the frequency of calculating the activation value and the weight histogram of the model (in the number of training rounds) is calculated. If set to 0, the histogram will not be computed. The verification data (or separated data) of histogram visualization must be clearly pointed out.
write_graph: Is it TensorBoard Visualization of images. If write_graph Set to True,Log files can become very large.
write_grads: Is it TensorBoard Visualized gradient histogram. histogram_freq Must be greater than 0.
batch_size: The input batch size of the afferent neuron network is calculated by histogram.
write_images: Is it TensorBoard The weight of the model is visualized with pictures.
embeddings_freq: The frequency at which the selected embedding layer will be saved (in the training wheel).
embeddings_layer_names: A list will be monitored by the name of the layer. If it is None Or empty lists, then all embedded layers are monitored.
embeddings_metadata: A dictionary, the name of the corresponding layer to the name of the metadata file that holds the embedded layer. See details about metadata data data formats. In case the same metadata is used in the embedding layer used, strings can be passed in.
embeddings_data: To be embedded in embeddings_layer_names Data for the specified layer. Numpy Array (if the model has a single input) or Numpy Array list (if the model has multiple inputs). Learn ore about embeddings. 
update_freq: 'batch' or 'epoch' Or integers. When used 'batch' When in each batch The loss and valuation are then written to TensorBoard Medium. The same applies to 'epoch' Medium. If an integer, such as 10,000, is used, the callback will write the loss and evaluation to every 10,000 samples. TensorBoard Medium. Note that it is frequently written to TensorBoard It will slow down your training.

5. How to record the training/verification loss/accuracy of each epoch?

Model.fit The function returns a History Callback, which has an attribute history Containing a package with continuous loss/Accurate lists. The code is as follows:

hist = model.fit(X, y,validation_split=0.2)  
print(hist.history)

Keras Output loss,val How to save these values in text 
Keras Medium fit The function returns a History Object, its History.history Attribute will store all the previous values in it. If there is a verification set, it will also include the changes of these indicators of the verification set.

hist=model.fit(train_set_x,train_set_y,batch_size=256,shuffle=True,nb_epoch=nb_epoch,validation_split=0.1)
with open('log_sgd_big_32.txt','w') as f:
    f.write(str(hist.history))

6. Multiple callback functions are separated by commas

For example:

model_weight_filepath = "./bert_classfition-test_model" + str(i) + ".weight"
earlystopping = EarlyStopping(monitor='val_acc', verbose=1, patience=3)
reducelronplateau = ReduceLROnPlateau(monitor="val_acc", verbose=1, mode='max', factor=0.5, patience=2)
checkpoint = ModelCheckpoint(filepath=model_weight_filepath, monitor='val_acc',
                             verbose=0, save_best_only=True, save_weights_only=True, mode='max', period=1)
model.fit_generator(
    train_D.__iter__(),
    steps_per_epoch=len(train_D),
    epochs=epochs,
    validation_data=valid_D.__iter__(),
    validation_steps=len(valid_D),
    callbacks=[earlystopping, reducelronplateau, checkpoint])

 

Tags: Attribute Python less pip

Posted on Fri, 06 Sep 2019 23:12:29 -0700 by kylera