7.1 言語モデルを使った文章生成

高速化のためにGPUを使います。

from ivory.common.context import np
np.context = 'gpu'

[1] 2019-06-20 22:45:26 (192ms) python3 (192ms)

PTBデータセットを読み出します。

from ivory.utils.repository import import_module

ptb = import_module("scratch2/dataset/ptb")
corpus, word_to_id, id_to_word = ptb.load_data("train")

[2] 2019-06-20 22:45:26 (14.0ms) python3 (206ms)

モデルを作成します。

from ivory.core.model import sequential

net = [
    ("input", 10000),
    ("embedding", 650),
    ("lstm", 650),
    ("lstm", 650),
    ("affine", 10000, "softmax_cross_entropy"),
]
model = sequential(net)

[3] 2019-06-20 22:45:26 (971ms) python3 (1.18s)

重みの共有をします。

em = model.layers[0]
affine = model.layers[-2]
affine.W.share_variable(em.W, transpose=True)
model.build()

[4] 2019-06-20 22:45:27 (15.6ms) python3 (1.19s)

<ivory.core.model.Model at 0x17fb832e0b8>

学習済みの重みを読み出します。

import os
import pickle
import ivory

directory = os.path.dirname(ivory.__file__)
directory = os.path.join(directory, "../docs/script")
with open(os.path.join(directory, 'better_rnnlm.pkl'), 'rb') as f:
    weights = pickle.load(f)

for v, weight in zip(model.weight_variables, weights):
    v.data = np.asarray(weight)

[5] 2019-06-20 22:45:27 (109ms) python3 (1.30s)

start文字とskip文字を設定します。

start_word = 'you'
start_id = word_to_id[start_word]
skip_words = ['N', '<unk>', '$']
skip_ids = [word_to_id[w] for w in skip_words]

[6] 2019-06-20 22:45:27 (15.6ms) python3 (1.32s)

Softmax関数を定義します。

def softmax(x):
    y = np.exp(x - x.max())
    return y / y.sum()

[7] 2019-06-20 22:45:27 (15.6ms) python3 (1.33s)

文章ジェネレータを定義します。

def generate(word_id, skip_ids=None):
    score = model.layers[-1].x
    yield id_to_word[word_id]
    while True:
        x = np.array(word_id).reshape(1, 1)
        model.set_data(x)
        model.forward(predict=True)
        p = softmax(score.d.flatten())
        sampled = np.random.choice(len(p), size=1, p=p)
        if skip_ids is None or sampled not in skip_ids:
            word_id = int(sampled)
            yield id_to_word[word_id]

[8] 2019-06-20 22:45:27 (5.97ms) python3 (1.34s)

文章を生成します。

sentences = []
for _ in range(5):
    gen = generate(start_id, skip_ids)
    words = [word for word, _ in zip(gen, range(100))]
    sentences.append(" ".join(words))

[9] 2019-06-20 22:45:27 (5.14s) python3 (6.48s)

  • you only support the course of the disaster <eos> that 's not only five weeks ago when the legislature has a couple of days well <eos> it 's the responsibility for declared by a new york school of public and political analysis <eos> mr. markey has all but the son of the cia <eos> while the two invented a commanding report himself in the 1930s spokesman said the report had worked out after of the o'kicki when there said that cathay 's administrative consultants are minimal <eos> in recent weeks farmers would use in the trade when the public needs

  • you and and the massachusetts law firm the office of management audit through the united states <eos> it also has been several years ago santa fe cities\/abc inc. said it obtained approval of certain charges of its legal and litigation to focus more on the closing too much more difficult to announce unsecured services <eos> the plan to merge with all of the units and sanford i. lee jr. an executive vice president in charge of the firm 's senior interest magazines montedison is joining a new syndicate <eos> the contract also seeks the power and other business ventures <eos>

  • you 're making a appeal with daniel b. barrett jr. as well as an admitted group to sell the plane <eos> amgen said the company expects output and revenue to improve revenue and the widely expected results <eos> valley pacific 's financial adviser and pricings in electric america was as great <eos> indeed any review will place this is among the company 's biggest buyers <eos> merabank will write off an ira banker said adjacent to the options <eos> but the bank has been attempting to press a plan to buy the remaining common operations in a leveraged transaction at

  • you be looking for a big role <eos> and since when the trust took steps to sell the company to move a stake in its bid to buy two shares of common stock will be redeemed at least or just a spokesman added <eos> late this year the company chose to cut cash on its other stock in check and may make an offer for the company from a labor-management group to take over the launched <eos> further debt may also be short-lived by the debt he said <eos> the minneapolis glass company for example reduced its interest in increase

  • you has recruited as a cautious price and in other financial markets the grand and office holder will receive m$ million or more <eos> but with real estate <eos> recently revco has soared to the highest price until the settlement in the first six months <eos> henry and other shareholder partners asked the union to change the price attempt to be converted the share of the new wealth under its shareholders or time with it <eos> plans to close a majority stake in the company to draw financing for its longtime vanguard partners is how the company 's needs to